[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:16 dist-test-slave-6zzp krb5kdc[2712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:16 dist-test-slave-6zzp krb5kdc[2712](info): set up 2 sockets
Feb 02 15:03:16 dist-test-slave-6zzp krb5kdc[2712](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260202 15:03:18.709295 2705 mini_kdc.cc:121] Time spent starting KDC: real 2.021s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:18 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:18.856297 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:43485
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:43485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:18.972468 2728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:18.972836 2728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:18.972944 2728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:18.976552 2728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:18.976691 2728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:18.976742 2728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:18.976806 2728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:18.976850 2728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:18.981585 2728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:43485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:18.982722 2728 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:18.983623 2728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260202 15:03:18.990478 2728 server_base.cc:1060] running on GCE node
W20260202 15:03:18.990509 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:18.990535 2733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:18.990660 2734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:18.991396 2728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:18.992439 2728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:18.993644 2728 hybrid_clock.cc:648] HybridClock initialized: now 1770044598993568 us; error 96 us; skew 500 ppm
Feb 02 15:03:18 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:18.996356 2728 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:18.997491 2728 webserver.cc:492] Webserver started at http://127.2.164.126:42751/ using document root <none> and password file <none>
I20260202 15:03:18.998008 2728 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:18.998055 2728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:18.998211 2728 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:18.999888 2728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "b0849a71cfcc41f49263246eee7afc5e"
format_stamp: "Formatted at 2026-02-02 15:03:18 on dist-test-slave-6zzp"
server_key: "2881bf48c556b2bc46acc102a7b02617"
server_key_iv: "b78d5fe09ee742015fece0bbfb2ed755"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.000358 2728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "b0849a71cfcc41f49263246eee7afc5e"
format_stamp: "Formatted at 2026-02-02 15:03:18 on dist-test-slave-6zzp"
server_key: "2881bf48c556b2bc46acc102a7b02617"
server_key_iv: "b78d5fe09ee742015fece0bbfb2ed755"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.003742 2728 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:19.005944 2743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.007001 2728 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:19.007125 2728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "b0849a71cfcc41f49263246eee7afc5e"
format_stamp: "Formatted at 2026-02-02 15:03:18 on dist-test-slave-6zzp"
server_key: "2881bf48c556b2bc46acc102a7b02617"
server_key_iv: "b78d5fe09ee742015fece0bbfb2ed755"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.007218 2728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:19.033113 2728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:19.036221 2728 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:19.036454 2728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:19.043962 2728 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:43485
I20260202 15:03:19.043972 2795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:43485 every 8 connection(s)
I20260202 15:03:19.045030 2728 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:19.048269 2796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:19.053297 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 2728
I20260202 15:03:19.053454 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:19.053777 2796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e: Bootstrap starting.
I20260202 15:03:19.054505 2705 external_mini_cluster.cc:1468] Setting key 02ab9562ef7c98966c86eb288d9a0c3d
I20260202 15:03:19.056062 2796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:19.056891 2796 log.cc:826] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:19.058995 2796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e: No bootstrap required, opened a new log
I20260202 15:03:19.062157 2796 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0849a71cfcc41f49263246eee7afc5e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43485 } }
I20260202 15:03:19.062414 2796 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:19.062469 2796 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0849a71cfcc41f49263246eee7afc5e, State: Initialized, Role: FOLLOWER
I20260202 15:03:19.062943 2796 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0849a71cfcc41f49263246eee7afc5e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43485 } }
I20260202 15:03:19.063107 2796 raft_consensus.cc:399] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:19.063189 2796 raft_consensus.cc:493] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:19.063283 2796 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:19.064316 2796 raft_consensus.cc:515] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0849a71cfcc41f49263246eee7afc5e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43485 } }
I20260202 15:03:19.064680 2796 leader_election.cc:304] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0849a71cfcc41f49263246eee7afc5e; no voters:
I20260202 15:03:19.064961 2796 leader_election.cc:290] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:19.065092 2801 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:19.065304 2801 raft_consensus.cc:697] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [term 1 LEADER]: Becoming Leader. State: Replica: b0849a71cfcc41f49263246eee7afc5e, State: Running, Role: LEADER
I20260202 15:03:19.065641 2801 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0849a71cfcc41f49263246eee7afc5e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43485 } }
I20260202 15:03:19.066162 2796 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:19.067417 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0849a71cfcc41f49263246eee7afc5e. Latest consensus state: current_term: 1 leader_uuid: "b0849a71cfcc41f49263246eee7afc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0849a71cfcc41f49263246eee7afc5e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43485 } } }
I20260202 15:03:19.067567 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:19.067821 2802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0849a71cfcc41f49263246eee7afc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0849a71cfcc41f49263246eee7afc5e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43485 } } }
I20260202 15:03:19.067911 2802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:19.068190 2805 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:19.071740 2805 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:19.077862 2805 catalog_manager.cc:1357] Generated new cluster ID: ba6db91793e14ecdb213ac3d6fdbc98f
I20260202 15:03:19.077941 2805 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:19.077883 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.056608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.2.164.126:43485)
0202 15:03:19.057089 (+ 481us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:19.057100 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:19.057692 (+ 592us) server_negotiation.cc:408] Connection header received
0202 15:03:19.061376 (+ 3684us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:19.061395 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:19.061678 (+ 283us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:19.061940 (+ 262us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:19.064223 (+ 2283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.065233 (+ 1010us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.066131 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.066408 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.070846 (+ 4438us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:19.070870 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:19.070883 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:19.070927 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:19.073304 (+ 2377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.074262 (+ 958us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.074272 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.074281 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.074463 (+ 182us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.074989 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.074993 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.074995 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.075617 (+ 622us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:19.075837 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:19.076696 (+ 859us) server_negotiation.cc:300] Negotiation successful
0202 15:03:19.076962 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260202 15:03:19.103009 2805 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:19.103953 2805 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:19.116607 2805 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b0849a71cfcc41f49263246eee7afc5e: Generated new TSK 0
I20260202 15:03:19.117475 2805 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:19.176796 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43485
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:19.285311 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:19.285537 2824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:19.285596 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:19.288820 2824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:19.288877 2824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:19.288957 2824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:19.293104 2824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:43485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:19.294060 2824 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:19.294957 2824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:19.301631 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:19.301622 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:19.301620 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:19.301867 2824 server_base.cc:1060] running on GCE node
I20260202 15:03:19.302345 2824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:19.302882 2824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:19.304052 2824 hybrid_clock.cc:648] HybridClock initialized: now 1770044599304025 us; error 41 us; skew 500 ppm
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:19.306767 2824 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:19.307785 2824 webserver.cc:492] Webserver started at http://127.2.164.65:45201/ using document root <none> and password file <none>
I20260202 15:03:19.308331 2824 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:19.308382 2824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:19.308596 2824 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:19.310397 2824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "ebca10503e7a4b03a20abb67a764b29c"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "7b9a39413a538d7dfd2097b5f4814411"
server_key_iv: "c827b64a9953a1b8cedf8686093d9eeb"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.310847 2824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "ebca10503e7a4b03a20abb67a764b29c"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "7b9a39413a538d7dfd2097b5f4814411"
server_key_iv: "c827b64a9953a1b8cedf8686093d9eeb"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.314435 2824 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:03:19.316614 2839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.317776 2824 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:03:19.317878 2824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "ebca10503e7a4b03a20abb67a764b29c"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "7b9a39413a538d7dfd2097b5f4814411"
server_key_iv: "c827b64a9953a1b8cedf8686093d9eeb"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.318015 2824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:19.334616 2824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:19.337309 2824 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:19.337522 2824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:19.338071 2824 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:19.338932 2824 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:19.338977 2824 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.339047 2824 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:19.339078 2824 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.348405 2824 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:38091
I20260202 15:03:19.348426 2952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:38091 every 8 connection(s)
I20260202 15:03:19.349416 2824 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:19.352516 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 2824
I20260202 15:03:19.352603 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:19.352833 2705 external_mini_cluster.cc:1468] Setting key 51b0136b1079a757d70abd9fdeab6e3b
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:19.362015 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.351343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50333 (local address 127.2.164.126:43485)
0202 15:03:19.351521 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:19.351525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:19.352166 (+ 641us) server_negotiation.cc:408] Connection header received
0202 15:03:19.353209 (+ 1043us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:19.353213 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:19.353276 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:19.353385 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:19.355018 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.355618 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.356315 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.356552 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.358680 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:19.358702 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:19.358705 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:19.358727 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:19.360120 (+ 1393us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.360646 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.360650 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.360653 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.360703 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.360981 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.360985 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.360987 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.361163 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:19.361286 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:19.361790 (+ 504us) server_negotiation.cc:300] Negotiation successful
0202 15:03:19.361894 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260202 15:03:19.362636 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.351586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:43485 (local address 127.2.164.65:50333)
0202 15:03:19.352010 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:19.352046 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:19.352951 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:19.353552 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:19.353560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:19.353945 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:19.354865 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:19.354878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.355744 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.355748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:19.356170 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:19.356177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.356357 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.356967 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:19.356991 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:19.358528 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:19.360278 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:19.360284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:19.360295 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:19.360539 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:19.360795 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:19.360799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:19.360802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:19.360886 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:19.361343 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:19.361348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:19.361559 (+ 211us) client_negotiation.cc:769] Sending connection context
0202 15:03:19.361725 (+ 166us) client_negotiation.cc:240] Negotiation successful
0202 15:03:19.361924 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:03:19.363850 2953 heartbeater.cc:344] Connected to a master server at 127.2.164.126:43485
I20260202 15:03:19.364135 2953 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:19.364765 2953 heartbeater.cc:507] Master 127.2.164.126:43485 requested a full tablet report, sending...
I20260202 15:03:19.366477 2760 ts_manager.cc:194] Registered new tserver with Master: ebca10503e7a4b03a20abb67a764b29c (127.2.164.65:38091)
I20260202 15:03:19.367729 2760 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:50333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:19.404419 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43485
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:19.512619 2960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:19.512850 2960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:19.512941 2960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:19.516122 2960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:19.516220 2960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:19.516346 2960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:19.520406 2960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:43485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:19.521450 2960 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:19.522307 2960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:19.528648 2968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:19.528671 2965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:19.528910 2960 server_base.cc:1060] running on GCE node
W20260202 15:03:19.528970 2966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:19.529366 2960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:19.529874 2960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:19.531016 2960 hybrid_clock.cc:648] HybridClock initialized: now 1770044599531007 us; error 23 us; skew 500 ppm
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:19.533686 2960 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:19.534667 2960 webserver.cc:492] Webserver started at http://127.2.164.66:44361/ using document root <none> and password file <none>
I20260202 15:03:19.535207 2960 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:19.535255 2960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:19.535450 2960 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:19.537213 2960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "0de00d7186a34eb3b48c79e8ab59e58b"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "2c62cb29da664f5f328a1955ae3fc3e5"
server_key_iv: "802aa30ac8d43395ee7fab5704c0425a"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.537729 2960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "0de00d7186a34eb3b48c79e8ab59e58b"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "2c62cb29da664f5f328a1955ae3fc3e5"
server_key_iv: "802aa30ac8d43395ee7fab5704c0425a"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.541249 2960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:03:19.543514 2975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.544669 2960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:19.544803 2960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "0de00d7186a34eb3b48c79e8ab59e58b"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "2c62cb29da664f5f328a1955ae3fc3e5"
server_key_iv: "802aa30ac8d43395ee7fab5704c0425a"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.544911 2960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:19.561226 2960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:19.563943 2960 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:19.564152 2960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:19.564721 2960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:19.565653 2960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:19.565727 2960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.565794 2960 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:19.565846 2960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.575984 2960 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:39277
I20260202 15:03:19.576018 3088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:39277 every 8 connection(s)
I20260202 15:03:19.576970 2960 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:19.579979 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 2960
I20260202 15:03:19.580056 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:19.580296 2705 external_mini_cluster.cc:1468] Setting key 0648e103f04c657518a0337f8415e9cf
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:03:19.590248 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.578806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42105 (local address 127.2.164.126:43485)
0202 15:03:19.578945 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:19.578948 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:19.579599 (+ 651us) server_negotiation.cc:408] Connection header received
0202 15:03:19.580520 (+ 921us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:19.580524 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:19.580587 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:19.580686 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:19.582652 (+ 1966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.583426 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.584101 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.584286 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.586846 (+ 2560us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:19.586869 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:19.586871 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:19.586894 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:19.588295 (+ 1401us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.588782 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.588786 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.588792 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.588833 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.589184 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.589188 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.589190 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.589340 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:19.589449 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:19.589931 (+ 482us) server_negotiation.cc:300] Negotiation successful
0202 15:03:19.590050 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260202 15:03:19.590843 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.579027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:43485 (local address 127.2.164.66:42105)
0202 15:03:19.579447 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:19.579484 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:19.580275 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:19.580892 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:19.580901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:19.581475 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:19.582494 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:19.582509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.583568 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.583572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:19.583976 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:19.583983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.584238 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.584930 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:19.584971 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:19.586698 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:19.588419 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:19.588424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:19.588435 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:19.588666 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:19.588934 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:19.588941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:19.588943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:19.589087 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:19.589460 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:19.589468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:19.589686 (+ 218us) client_negotiation.cc:769] Sending connection context
0202 15:03:19.589883 (+ 197us) client_negotiation.cc:240] Negotiation successful
0202 15:03:19.590133 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:03:19.591910 3089 heartbeater.cc:344] Connected to a master server at 127.2.164.126:43485
I20260202 15:03:19.592149 3089 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:19.592765 3089 heartbeater.cc:507] Master 127.2.164.126:43485 requested a full tablet report, sending...
I20260202 15:03:19.594365 2760 ts_manager.cc:194] Registered new tserver with Master: 0de00d7186a34eb3b48c79e8ab59e58b (127.2.164.66:39277)
I20260202 15:03:19.595167 2760 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:42105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:19.632378 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43485
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:19.740319 3096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:19.740528 3096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:19.740625 3096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:19.743903 3096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:19.743968 3096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:19.744094 3096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:19.748416 3096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:43485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:19.749629 3096 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:19.750523 3096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:19.757525 3102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:19.757596 3104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:19.757632 3101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:19.758136 3096 server_base.cc:1060] running on GCE node
I20260202 15:03:19.758518 3096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:19.759132 3096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:19.760336 3096 hybrid_clock.cc:648] HybridClock initialized: now 1770044599760311 us; error 42 us; skew 500 ppm
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:19.763197 3096 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:19.764361 3096 webserver.cc:492] Webserver started at http://127.2.164.67:38245/ using document root <none> and password file <none>
I20260202 15:03:19.764889 3096 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:19.764941 3096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:19.765180 3096 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:19.766918 3096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "90d0344374ba45c3864a71266f8a1eab"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "cfae3a583f5348315caffabf5058e0a4"
server_key_iv: "f784cf89ed085f4afb39720e2c30c8ea"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.767503 3096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "90d0344374ba45c3864a71266f8a1eab"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "cfae3a583f5348315caffabf5058e0a4"
server_key_iv: "f784cf89ed085f4afb39720e2c30c8ea"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.771173 3096 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:19.773449 3111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.774482 3096 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:19.774578 3096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "90d0344374ba45c3864a71266f8a1eab"
format_stamp: "Formatted at 2026-02-02 15:03:19 on dist-test-slave-6zzp"
server_key: "cfae3a583f5348315caffabf5058e0a4"
server_key_iv: "f784cf89ed085f4afb39720e2c30c8ea"
server_key_version: "encryptionkey@0"
I20260202 15:03:19.774658 3096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:19.787477 3096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:19.790495 3096 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:19.790639 3096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:19.791123 3096 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:19.791957 3096 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:19.792003 3096 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.792043 3096 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:19.792058 3096 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:19.801991 3096 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:41087
I20260202 15:03:19.802012 3224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:41087 every 8 connection(s)
I20260202 15:03:19.802939 3096 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:19.807952 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3096
I20260202 15:03:19.808036 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:19.808301 2705 external_mini_cluster.cc:1468] Setting key e58410721579621b7685d0957a72ca8e
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:19.815743 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.804779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:34389 (local address 127.2.164.126:43485)
0202 15:03:19.804920 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:19.804924 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:19.805666 (+ 742us) server_negotiation.cc:408] Connection header received
0202 15:03:19.806435 (+ 769us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:19.806439 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:19.806494 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:19.806568 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:19.808214 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.808786 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.810103 (+ 1317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.810276 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.812504 (+ 2228us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:19.812518 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:19.812520 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:19.812544 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:19.814005 (+ 1461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.814456 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.814459 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.814461 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.814500 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.814771 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.814774 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.814775 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.814942 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:19.815029 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:19.815498 (+ 469us) server_negotiation.cc:300] Negotiation successful
0202 15:03:19.815605 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:03:19.816392 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.805086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:43485 (local address 127.2.164.67:34389)
0202 15:03:19.805520 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:19.805557 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:19.806248 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:19.806718 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:19.806725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:19.807100 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:19.808055 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:19.808073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.808910 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.808914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:19.809995 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:19.810005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.810223 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.810857 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:19.810882 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:19.812376 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:19.814111 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:19.814115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:19.814124 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:19.814363 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:19.814581 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:19.814584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:19.814585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:19.814665 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:19.815036 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:19.815041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:19.815265 (+ 224us) client_negotiation.cc:769] Sending connection context
0202 15:03:19.815447 (+ 182us) client_negotiation.cc:240] Negotiation successful
0202 15:03:19.815660 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":126,"threads_started":1}
I20260202 15:03:19.817540 3225 heartbeater.cc:344] Connected to a master server at 127.2.164.126:43485
I20260202 15:03:19.817762 3225 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:19.818239 3225 heartbeater.cc:507] Master 127.2.164.126:43485 requested a full tablet report, sending...
I20260202 15:03:19.819263 2760 ts_manager.cc:194] Registered new tserver with Master: 90d0344374ba45c3864a71266f8a1eab (127.2.164.67:41087)
I20260202 15:03:19.819864 2760 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:34389
I20260202 15:03:19.822714 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:19.846678 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.838290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.2.164.65:38091)
0202 15:03:19.838582 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:19.838587 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:19.838612 (+ 25us) server_negotiation.cc:408] Connection header received
0202 15:03:19.838664 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:19.838669 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:19.838853 (+ 184us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:19.838988 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:19.839714 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.840410 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.841137 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.841356 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.843384 (+ 2028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:19.843407 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:19.843419 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:19.843443 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:19.845165 (+ 1722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.845563 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.845567 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.845568 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.845618 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.845812 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.845814 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.845815 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.845993 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:19.846142 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:19.846323 (+ 181us) server_negotiation.cc:300] Negotiation successful
0202 15:03:19.846490 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
W20260202 15:03:19.847591 2867 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54480
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:19.871564 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:19.863861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.2.164.65:38091)
0202 15:03:19.864014 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:19.864018 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:19.864050 (+ 32us) server_negotiation.cc:408] Connection header received
0202 15:03:19.864166 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:19.864169 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:19.864247 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:19.864316 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:19.865502 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.865995 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:19.866618 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:19.866792 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:19.868967 (+ 2175us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:19.868985 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:19.868988 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:19.869055 (+ 67us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:19.870277 (+ 1222us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.870619 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.870621 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.870623 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.870660 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:19.870840 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:19.870842 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:19.870843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:19.870966 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:19.871069 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:19.871254 (+ 185us) server_negotiation.cc:300] Negotiation successful
0202 15:03:19.871377 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260202 15:03:19.872751 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 2824
I20260202 15:03:19.879436 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 2960
I20260202 15:03:19.887687 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3096
I20260202 15:03:19.893738 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 2728
2026-02-02T15:03:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3234 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[3239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[3239](info): set up 2 sockets
Feb 02 15:03:19 dist-test-slave-6zzp krb5kdc[3239](info): commencing operation
krb5kdc: starting...
W20260202 15:03:21.913085 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:21 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:22.055977 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38249
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:22.163020 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:22.163300 3255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:22.163398 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:22.167327 3255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:22.167416 3255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:22.167464 3255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:22.167506 3255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:22.167548 3255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:22.172219 3255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:22.173421 3255 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:22.174317 3255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:22.180258 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.180259 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.180227 3263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:22.180486 3255 server_base.cc:1060] running on GCE node
I20260202 15:03:22.181062 3255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:22.181941 3255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:22.183095 3255 hybrid_clock.cc:648] HybridClock initialized: now 1770044602183072 us; error 40 us; skew 500 ppm
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:22.185672 3255 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:22.186731 3255 webserver.cc:492] Webserver started at http://127.2.164.126:35641/ using document root <none> and password file <none>
I20260202 15:03:22.187228 3255 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:22.187273 3255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:22.187424 3255 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:22.189105 3255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "b88e2844e4c14452aa354ab44f612583"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "2ad40ce31fd12dc1957103366255510b"
server_key_iv: "0aaadc237b50a3a31ab4c41c37c30a4d"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.189540 3255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "b88e2844e4c14452aa354ab44f612583"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "2ad40ce31fd12dc1957103366255510b"
server_key_iv: "0aaadc237b50a3a31ab4c41c37c30a4d"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.192814 3255 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260202 15:03:22.195083 3270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.196193 3255 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:22.196321 3255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "b88e2844e4c14452aa354ab44f612583"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "2ad40ce31fd12dc1957103366255510b"
server_key_iv: "0aaadc237b50a3a31ab4c41c37c30a4d"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.196426 3255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:22.217805 3255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:22.225020 3255 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:22.225245 3255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:22.232772 3322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:38249 every 8 connection(s)
I20260202 15:03:22.232777 3255 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38249
I20260202 15:03:22.233850 3255 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:22.236557 3323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:22.241837 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3255
I20260202 15:03:22.241932 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:22.242163 2705 external_mini_cluster.cc:1468] Setting key 00fe26c935fb07ebbf5b291c487f7b21
I20260202 15:03:22.242341 3323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583: Bootstrap starting.
I20260202 15:03:22.244627 3323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:22.245544 3323 log.cc:826] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:22.247471 3323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583: No bootstrap required, opened a new log
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:22.250532 3323 raft_consensus.cc:359] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88e2844e4c14452aa354ab44f612583" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38249 } }
I20260202 15:03:22.250707 3323 raft_consensus.cc:385] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:22.250771 3323 raft_consensus.cc:740] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88e2844e4c14452aa354ab44f612583, State: Initialized, Role: FOLLOWER
I20260202 15:03:22.251204 3323 consensus_queue.cc:260] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88e2844e4c14452aa354ab44f612583" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38249 } }
I20260202 15:03:22.251312 3323 raft_consensus.cc:399] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:22.251431 3323 raft_consensus.cc:493] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:22.251535 3323 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:22.252571 3323 raft_consensus.cc:515] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88e2844e4c14452aa354ab44f612583" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38249 } }
I20260202 15:03:22.252988 3323 leader_election.cc:304] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b88e2844e4c14452aa354ab44f612583; no voters:
I20260202 15:03:22.253353 3323 leader_election.cc:290] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:22.253746 3328 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:22.253926 3328 raft_consensus.cc:697] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [term 1 LEADER]: Becoming Leader. State: Replica: b88e2844e4c14452aa354ab44f612583, State: Running, Role: LEADER
I20260202 15:03:22.254204 3328 consensus_queue.cc:237] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88e2844e4c14452aa354ab44f612583" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38249 } }
I20260202 15:03:22.254567 3323 sys_catalog.cc:565] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:22.255803 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:22.243430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.2.164.126:38249)
0202 15:03:22.243811 (+ 381us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:22.243819 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:22.243852 (+ 33us) server_negotiation.cc:408] Connection header received
0202 15:03:22.244351 (+ 499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:22.244367 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:22.244646 (+ 279us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:22.244954 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:22.246016 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.246870 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:22.247698 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.247894 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:22.250310 (+ 2416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:22.250326 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:22.250336 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:22.250370 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:22.252546 (+ 2176us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:22.252968 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:22.252973 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:22.252978 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:22.253112 (+ 134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:22.253384 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:22.253387 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:22.253389 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:22.253850 (+ 461us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:22.254056 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:22.254775 (+ 719us) server_negotiation.cc:300] Negotiation successful
0202 15:03:22.254974 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":135,"threads_started":1}
I20260202 15:03:22.255833 3329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b88e2844e4c14452aa354ab44f612583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88e2844e4c14452aa354ab44f612583" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38249 } } }
I20260202 15:03:22.255839 3330 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b88e2844e4c14452aa354ab44f612583. Latest consensus state: current_term: 1 leader_uuid: "b88e2844e4c14452aa354ab44f612583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88e2844e4c14452aa354ab44f612583" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38249 } } }
I20260202 15:03:22.256306 3330 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:22.256261 3329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:22.256769 3337 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:22.259912 3337 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:22.265128 3337 catalog_manager.cc:1357] Generated new cluster ID: 05c7644fa6a9454095f0f0f43e28b4f3
I20260202 15:03:22.265210 3337 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:22.285840 3337 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:22.286803 3337 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:22.295519 3337 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b88e2844e4c14452aa354ab44f612583: Generated new TSK 0
I20260202 15:03:22.296389 3337 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:22.361166 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38249
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:22.475278 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:22.475531 3351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:22.475594 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:22.479152 3351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:22.479229 3351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:22.479315 3351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:22.483623 3351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:38249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:22.484741 3351 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:22.485632 3351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260202 15:03:22.492926 3351 server_base.cc:1060] running on GCE node
W20260202 15:03:22.492878 3356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.492877 3357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.492878 3359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:22.493851 3351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:22.494426 3351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:22.495608 3351 hybrid_clock.cc:648] HybridClock initialized: now 1770044602495582 us; error 44 us; skew 500 ppm
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:22.498282 3351 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:22.499329 3351 webserver.cc:492] Webserver started at http://127.2.164.65:40987/ using document root <none> and password file <none>
I20260202 15:03:22.499910 3351 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:22.499990 3351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:22.500236 3351 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:22.502080 3351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "f2e1df8e4d28440099808bc74a908af6"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "ae8d3ba2e91c19c461acfd58489a8172"
server_key_iv: "7828484cc7aa8ae102510ddc5f68791f"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.502571 3351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e1df8e4d28440099808bc74a908af6"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "ae8d3ba2e91c19c461acfd58489a8172"
server_key_iv: "7828484cc7aa8ae102510ddc5f68791f"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.505982 3351 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:03:22.508134 3366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.509248 3351 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:22.509378 3351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "f2e1df8e4d28440099808bc74a908af6"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "ae8d3ba2e91c19c461acfd58489a8172"
server_key_iv: "7828484cc7aa8ae102510ddc5f68791f"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.509491 3351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:22.522585 3351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:22.525249 3351 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:22.525460 3351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:22.526057 3351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:22.526959 3351 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:22.527035 3351 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.527100 3351 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:22.527153 3351 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.537714 3351 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:41645
I20260202 15:03:22.537746 3479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:41645 every 8 connection(s)
I20260202 15:03:22.538750 3351 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:22.547179 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3351
I20260202 15:03:22.547324 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:22.547616 2705 external_mini_cluster.cc:1468] Setting key 84a71188c33633ee4b86d77262b0ab58
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:22.554747 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:22.540690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36341 (local address 127.2.164.126:38249)
0202 15:03:22.540907 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:22.540913 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:22.541876 (+ 963us) server_negotiation.cc:408] Connection header received
0202 15:03:22.543132 (+ 1256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:22.543136 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:22.543211 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:22.543313 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:22.545707 (+ 2394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.546783 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:22.547853 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.548036 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:22.551044 (+ 3008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:22.551061 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:22.551064 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:22.551088 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:22.552475 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:22.553067 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:22.553073 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:22.553078 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:22.553134 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:22.553420 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:22.553426 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:22.553431 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:22.553648 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:22.553756 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:22.554334 (+ 578us) server_negotiation.cc:300] Negotiation successful
0202 15:03:22.554476 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260202 15:03:22.555202 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:22.541039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38249 (local address 127.2.164.65:36341)
0202 15:03:22.541682 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:22.541737 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:22.542848 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:22.543539 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:22.543553 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:22.544152 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:22.545534 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:22.545568 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.546943 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:22.546951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:22.547691 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:22.547704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.547975 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:22.548941 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:22.548977 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:22.550882 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:22.552613 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:22.552621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:22.552634 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:22.552920 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:22.553225 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:22.553227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:22.553229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:22.553315 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:22.553753 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:22.553758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:22.554059 (+ 301us) client_negotiation.cc:769] Sending connection context
0202 15:03:22.554230 (+ 171us) client_negotiation.cc:240] Negotiation successful
0202 15:03:22.554446 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":175,"threads_started":1}
I20260202 15:03:22.556550 3480 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38249
I20260202 15:03:22.556813 3480 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:22.557442 3480 heartbeater.cc:507] Master 127.2.164.126:38249 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:22.559024 3286 ts_manager.cc:194] Registered new tserver with Master: f2e1df8e4d28440099808bc74a908af6 (127.2.164.65:41645)
I20260202 15:03:22.560194 3286 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:36341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:22.605176 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38249
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:22.714967 3487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:22.715200 3487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:22.715260 3487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:22.718459 3487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:22.718515 3487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:22.718595 3487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:22.722695 3487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:38249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:22.723729 3487 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:22.724545 3487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:22.731355 3495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:22.731420 3487 server_base.cc:1060] running on GCE node
W20260202 15:03:22.731352 3492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.731429 3493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:22.731926 3487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:22.732458 3487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:22.733601 3487 hybrid_clock.cc:648] HybridClock initialized: now 1770044602733577 us; error 40 us; skew 500 ppm
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:22.736124 3487 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:22.737282 3487 webserver.cc:492] Webserver started at http://127.2.164.66:35823/ using document root <none> and password file <none>
I20260202 15:03:22.737860 3487 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:22.737910 3487 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:22.738112 3487 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:22.739828 3487 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "05592cf885b925f17432e24b5acaf541"
server_key_iv: "40338f6a04d9ec7c20900af35b1a768e"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.740370 3487 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "05592cf885b925f17432e24b5acaf541"
server_key_iv: "40338f6a04d9ec7c20900af35b1a768e"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.743844 3487 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260202 15:03:22.746124 3502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.747202 3487 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260202 15:03:22.747336 3487 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "05592cf885b925f17432e24b5acaf541"
server_key_iv: "40338f6a04d9ec7c20900af35b1a768e"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.747448 3487 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:22.762439 3487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:22.765893 3487 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:22.766120 3487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:22.766782 3487 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:22.767728 3487 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:22.767804 3487 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.767871 3487 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:22.767921 3487 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.777776 3487 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:42635
I20260202 15:03:22.777791 3615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:42635 every 8 connection(s)
I20260202 15:03:22.778831 3487 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:22.781185 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3487
I20260202 15:03:22.781404 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:22.781692 2705 external_mini_cluster.cc:1468] Setting key 2f7306d2af930fdb5e18c86170e0df6b
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:03:22.794203 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:22.780775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:36029 (local address 127.2.164.126:38249)
0202 15:03:22.781010 (+ 235us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:22.781018 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:22.781917 (+ 899us) server_negotiation.cc:408] Connection header received
0202 15:03:22.782917 (+ 1000us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:22.782921 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:22.782976 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:22.783090 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:22.785456 (+ 2366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.786280 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:22.787033 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.787267 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:22.790220 (+ 2953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:22.790241 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:22.790244 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:22.790272 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:22.791868 (+ 1596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:22.792413 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:22.792416 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:22.792417 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:22.792466 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:22.792879 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:22.792882 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:22.792883 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:22.793131 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:22.793220 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:22.793860 (+ 640us) server_negotiation.cc:300] Negotiation successful
0202 15:03:22.793978 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260202 15:03:22.794906 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:22.781185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38249 (local address 127.2.164.66:36029)
0202 15:03:22.781757 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:22.781802 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:22.782706 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:22.783301 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:22.783312 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:22.783921 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:22.785284 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:22.785299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.786458 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:22.786461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:22.786895 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:22.786903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:22.787085 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:22.787833 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:22.787861 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:22.790038 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:22.792023 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:22.792028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:22.792039 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:22.792287 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:22.792555 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:22.792557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:22.792559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:22.792794 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:22.793269 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:22.793276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:22.793550 (+ 274us) client_negotiation.cc:769] Sending connection context
0202 15:03:22.793744 (+ 194us) client_negotiation.cc:240] Negotiation successful
0202 15:03:22.793982 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:03:22.796429 3616 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38249
I20260202 15:03:22.796705 3616 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:22.797356 3616 heartbeater.cc:507] Master 127.2.164.126:38249 requested a full tablet report, sending...
I20260202 15:03:22.798564 3286 ts_manager.cc:194] Registered new tserver with Master: ca08a4a9ecf6452dbb02513b5c21bdc3 (127.2.164.66:42635)
I20260202 15:03:22.799221 3286 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:36029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:22.834013 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38249
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:22.942906 3623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:22.943217 3623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:22.943316 3623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:22.946698 3623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:22.946816 3623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:22.946935 3623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:22.951432 3623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:38249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:22.952630 3623 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:22.953568 3623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:22.960376 3628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.960443 3629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:22.960446 3631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:22.960847 3623 server_base.cc:1060] running on GCE node
I20260202 15:03:22.961241 3623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:22.961762 3623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:22.962939 3623 hybrid_clock.cc:648] HybridClock initialized: now 1770044602962913 us; error 41 us; skew 500 ppm
Feb 02 15:03:22 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:22.965811 3623 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:22.966892 3623 webserver.cc:492] Webserver started at http://127.2.164.67:35041/ using document root <none> and password file <none>
I20260202 15:03:22.967464 3623 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:22.967515 3623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:22.967734 3623 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:22.969550 3623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "c17ce29b56d643118b504b0320bcd7b5"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "004f845c57eafdc19ae8fdc9934b3d1f"
server_key_iv: "1c947703c513b1530a20973c6e8e9340"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.970054 3623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "c17ce29b56d643118b504b0320bcd7b5"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "004f845c57eafdc19ae8fdc9934b3d1f"
server_key_iv: "1c947703c513b1530a20973c6e8e9340"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.973600 3623 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:22.975792 3638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:22.976974 3623 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:22.977216 3623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "c17ce29b56d643118b504b0320bcd7b5"
format_stamp: "Formatted at 2026-02-02 15:03:22 on dist-test-slave-6zzp"
server_key: "004f845c57eafdc19ae8fdc9934b3d1f"
server_key_iv: "1c947703c513b1530a20973c6e8e9340"
server_key_version: "encryptionkey@0"
I20260202 15:03:22.977358 3623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:23.003963 3623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:23.007236 3623 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:23.007462 3623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:23.008049 3623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:23.008961 3623 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:23.009063 3623 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:23.009160 3623 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:23.009217 3623 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:23.019333 3623 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:36603
I20260202 15:03:23.019367 3751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:36603 every 8 connection(s)
I20260202 15:03:23.020326 3623 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:23.030174 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3623
I20260202 15:03:23.030303 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:23.030558 2705 external_mini_cluster.cc:1468] Setting key 2a65ae767dc0d7ebb0c2d7e3b9611735
I20260202 15:03:23.033187 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.022088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:35001 (local address 127.2.164.126:38249)
0202 15:03:23.022245 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.022249 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.022945 (+ 696us) server_negotiation.cc:408] Connection header received
0202 15:03:23.023803 (+ 858us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.023807 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.023866 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.023979 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:23.025713 (+ 1734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.026215 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.026910 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.027092 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.029409 (+ 2317us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:23.029435 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:23.029438 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:23.029464 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:23.031071 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.031660 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.031664 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.031666 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.031722 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.032052 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.032056 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.032058 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.032219 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:23.032332 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.032893 (+ 561us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.033062 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260202 15:03:23.033815 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.022324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38249 (local address 127.2.164.67:35001)
0202 15:03:23.022787 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:23.022823 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:23.023603 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:23.024134 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:23.024142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:23.024547 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:23.025557 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.025572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.026367 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.026370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:23.026788 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.026795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.027040 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.027683 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:23.027708 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:23.029232 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:23.031208 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:23.031215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:23.031225 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:23.031552 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:23.031827 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:23.031831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:23.031833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:23.031946 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:23.032331 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:23.032339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:23.032566 (+ 227us) client_negotiation.cc:769] Sending connection context
0202 15:03:23.032829 (+ 263us) client_negotiation.cc:240] Negotiation successful
0202 15:03:23.033099 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":106,"threads_started":1}
I20260202 15:03:23.034895 3752 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38249
I20260202 15:03:23.035130 3752 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:23.035627 3752 heartbeater.cc:507] Master 127.2.164.126:38249 requested a full tablet report, sending...
I20260202 15:03:23.036717 3286 ts_manager.cc:194] Registered new tserver with Master: c17ce29b56d643118b504b0320bcd7b5 (127.2.164.67:36603)
I20260202 15:03:23.037385 3286 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:35001
I20260202 15:03:23.044538 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260202 15:03:23.054121 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.047659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.2.164.126:38249)
0202 15:03:23.047811 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.047815 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.047940 (+ 125us) server_negotiation.cc:408] Connection header received
0202 15:03:23.048036 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.048039 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.048082 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.048162 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:23.048985 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.049547 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.050187 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.050327 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.051377 (+ 1050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:23.051391 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:23.051393 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:23.051415 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:23.052710 (+ 1295us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.053111 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.053113 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.053115 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.053156 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.053356 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.053358 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.053359 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.053522 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:23.053626 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.053903 (+ 277us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.053998 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260202 15:03:23.058491 3286 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:23.060774 3286 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:23.073809 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.069080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:42635 (local address 127.0.0.1:60946)
0202 15:03:23.069429 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:23.069469 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:23.069609 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:23.070487 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:23.070493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:23.070516 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:23.070835 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.070843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.072437 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.072440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:23.073413 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.073423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.073538 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.073555 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:03:23.073606 (+ 51us) client_negotiation.cc:240] Negotiation successful
0202 15:03:23.073662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260202 15:03:23.074680 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.069951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.2.164.67:36603)
0202 15:03:23.070286 (+ 335us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.070292 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.070313 (+ 21us) server_negotiation.cc:408] Connection header received
0202 15:03:23.070397 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.070404 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.070562 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.071073 (+ 511us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:23.071209 (+ 136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.072313 (+ 1104us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.073888 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.074376 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.074457 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.074507 (+ 50us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.074570 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20260202 15:03:23.074679 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.070305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.2.164.65:41645)
0202 15:03:23.070810 (+ 505us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.070816 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.070837 (+ 21us) server_negotiation.cc:408] Connection header received
0202 15:03:23.070923 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.070928 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.071100 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.071227 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:23.071585 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.072271 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.073743 (+ 1472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.074286 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.074410 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.074470 (+ 60us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.074553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":230,"threads_started":1}
I20260202 15:03:23.073812 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.069634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:36603 (local address 127.0.0.1:36026)
0202 15:03:23.069985 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:23.069999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:23.070090 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:23.070729 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:23.070733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:23.070749 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:23.070953 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.070959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.072481 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.072483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:23.073261 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.073272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.073389 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.073449 (+ 60us) client_negotiation.cc:769] Sending connection context
0202 15:03:23.073544 (+ 95us) client_negotiation.cc:240] Negotiation successful
0202 15:03:23.073651 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
I20260202 15:03:23.075138 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.069628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41645 (local address 127.0.0.1:55164)
0202 15:03:23.070270 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:23.070283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:23.070371 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:23.071232 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:23.071235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:23.071252 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:23.071465 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.071471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.072397 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.072402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:23.073459 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.073469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.073548 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.073558 (+ 10us) client_negotiation.cc:769] Sending connection context
0202 15:03:23.073606 (+ 48us) client_negotiation.cc:240] Negotiation successful
0202 15:03:23.073662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":105,"threads_started":1}
I20260202 15:03:23.075181 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.069232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.2.164.66:42635)
0202 15:03:23.069988 (+ 756us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.069993 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.070012 (+ 19us) server_negotiation.cc:408] Connection header received
0202 15:03:23.070076 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.070081 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.070287 (+ 206us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.070973 (+ 686us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:23.071119 (+ 146us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.072279 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.074270 (+ 1991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.074785 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.074882 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.074941 (+ 59us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.075030 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":112,"threads_started":1}
I20260202 15:03:23.077661 3550 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce6b185ca1d47548cc87040fcd12465 (DEFAULT_TABLE table=test-table [id=060b032d6256423784a6fdc4f8546d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:23.077723 3686 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce6b185ca1d47548cc87040fcd12465 (DEFAULT_TABLE table=test-table [id=060b032d6256423784a6fdc4f8546d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:23.077723 3414 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce6b185ca1d47548cc87040fcd12465 (DEFAULT_TABLE table=test-table [id=060b032d6256423784a6fdc4f8546d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:23.078612 3414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6b185ca1d47548cc87040fcd12465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:23.078619 3550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6b185ca1d47548cc87040fcd12465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:23.078612 3686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6b185ca1d47548cc87040fcd12465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:23.084056 3769 tablet_bootstrap.cc:492] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: Bootstrap starting.
I20260202 15:03:23.085246 3770 tablet_bootstrap.cc:492] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: Bootstrap starting.
I20260202 15:03:23.085490 3771 tablet_bootstrap.cc:492] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: Bootstrap starting.
I20260202 15:03:23.086866 3769 tablet_bootstrap.cc:654] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:23.087080 3770 tablet_bootstrap.cc:654] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:23.087785 3769 log.cc:826] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:23.087916 3771 tablet_bootstrap.cc:654] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:23.087867 3770 log.cc:826] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:23.088682 3771 log.cc:826] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:23.089771 3769 tablet_bootstrap.cc:492] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: No bootstrap required, opened a new log
I20260202 15:03:23.089812 3770 tablet_bootstrap.cc:492] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: No bootstrap required, opened a new log
I20260202 15:03:23.089984 3769 ts_tablet_manager.cc:1403] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:03:23.089987 3770 ts_tablet_manager.cc:1403] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:03:23.090327 3771 tablet_bootstrap.cc:492] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: No bootstrap required, opened a new log
I20260202 15:03:23.090497 3771 ts_tablet_manager.cc:1403] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:03:23.093163 3769 raft_consensus.cc:359] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.093186 3770 raft_consensus.cc:359] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.093380 3770 raft_consensus.cc:385] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:23.093415 3769 raft_consensus.cc:385] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:23.093459 3770 raft_consensus.cc:740] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e1df8e4d28440099808bc74a908af6, State: Initialized, Role: FOLLOWER
I20260202 15:03:23.093490 3769 raft_consensus.cc:740] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca08a4a9ecf6452dbb02513b5c21bdc3, State: Initialized, Role: FOLLOWER
I20260202 15:03:23.093845 3771 raft_consensus.cc:359] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.093925 3770 consensus_queue.cc:260] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.093923 3769 consensus_queue.cc:260] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.094025 3771 raft_consensus.cc:385] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:23.094074 3771 raft_consensus.cc:740] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17ce29b56d643118b504b0320bcd7b5, State: Initialized, Role: FOLLOWER
I20260202 15:03:23.094496 3771 consensus_queue.cc:260] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.094592 3480 heartbeater.cc:499] Master 127.2.164.126:38249 was elected leader, sending a full tablet report...
I20260202 15:03:23.094815 3769 ts_tablet_manager.cc:1434] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:23.094820 3770 ts_tablet_manager.cc:1434] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:23.095130 3616 heartbeater.cc:499] Master 127.2.164.126:38249 was elected leader, sending a full tablet report...
I20260202 15:03:23.095211 3752 heartbeater.cc:499] Master 127.2.164.126:38249 was elected leader, sending a full tablet report...
I20260202 15:03:23.095278 3771 ts_tablet_manager.cc:1434] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:23.248091 3777 raft_consensus.cc:493] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:23.248319 3777 raft_consensus.cc:515] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.249748 3777 leader_election.cc:290] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca08a4a9ecf6452dbb02513b5c21bdc3 (127.2.164.66:42635), f2e1df8e4d28440099808bc74a908af6 (127.2.164.65:41645)
I20260202 15:03:23.253170 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.249962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41645 (local address 127.2.164.67:59987)
0202 15:03:23.250607 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:23.250646 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:23.250734 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:23.251013 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:23.251017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:23.251035 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:23.251287 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.251292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.252135 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.252138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:23.252721 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.252728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.252862 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.252885 (+ 23us) client_negotiation.cc:769] Sending connection context
0202 15:03:23.252938 (+ 53us) client_negotiation.cc:240] Negotiation successful
0202 15:03:23.252996 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":10,"thread_start_us":109,"threads_started":1}
I20260202 15:03:23.253680 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.250500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59987 (local address 127.2.164.65:41645)
0202 15:03:23.250666 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.250670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.250688 (+ 18us) server_negotiation.cc:408] Connection header received
0202 15:03:23.250842 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.250845 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.250896 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.250979 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:23.251414 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.251989 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.252846 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.253365 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.253414 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.253471 (+ 57us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.253539 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260202 15:03:23.253371 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.249932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:42635 (local address 127.2.164.67:56937)
0202 15:03:23.250053 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:23.250086 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:23.250209 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:23.250740 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:23.250743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:23.250759 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:23.251022 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.251029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.252115 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.252119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:23.253034 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:23.253043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.253145 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.253162 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:03:23.253207 (+ 45us) client_negotiation.cc:240] Negotiation successful
0202 15:03:23.253261 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260202 15:03:23.254340 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.250236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56937 (local address 127.2.164.66:42635)
0202 15:03:23.250411 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.250419 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.250435 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:03:23.250489 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.250492 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.250542 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.250619 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:23.251127 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.251979 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.253453 (+ 1474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.253937 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.253971 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.254017 (+ 46us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.254066 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260202 15:03:23.254429 3434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce6b185ca1d47548cc87040fcd12465" candidate_uuid: "c17ce29b56d643118b504b0320bcd7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e1df8e4d28440099808bc74a908af6" is_pre_election: true
I20260202 15:03:23.254702 3434 raft_consensus.cc:2468] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c17ce29b56d643118b504b0320bcd7b5 in term 0.
I20260202 15:03:23.254838 3570 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce6b185ca1d47548cc87040fcd12465" candidate_uuid: "c17ce29b56d643118b504b0320bcd7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" is_pre_election: true
I20260202 15:03:23.255069 3570 raft_consensus.cc:2468] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c17ce29b56d643118b504b0320bcd7b5 in term 0.
I20260202 15:03:23.255198 3642 leader_election.cc:304] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17ce29b56d643118b504b0320bcd7b5, f2e1df8e4d28440099808bc74a908af6; no voters:
I20260202 15:03:23.255491 3777 raft_consensus.cc:2804] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:23.255564 3777 raft_consensus.cc:493] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:23.255615 3777 raft_consensus.cc:3060] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:23.256556 3777 raft_consensus.cc:515] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.256999 3777 leader_election.cc:290] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [CANDIDATE]: Term 1 election: Requested vote from peers ca08a4a9ecf6452dbb02513b5c21bdc3 (127.2.164.66:42635), f2e1df8e4d28440099808bc74a908af6 (127.2.164.65:41645)
I20260202 15:03:23.257349 3570 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce6b185ca1d47548cc87040fcd12465" candidate_uuid: "c17ce29b56d643118b504b0320bcd7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3"
I20260202 15:03:23.257397 3434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce6b185ca1d47548cc87040fcd12465" candidate_uuid: "c17ce29b56d643118b504b0320bcd7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e1df8e4d28440099808bc74a908af6"
I20260202 15:03:23.257491 3570 raft_consensus.cc:3060] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:23.257524 3434 raft_consensus.cc:3060] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:23.258395 3570 raft_consensus.cc:2468] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c17ce29b56d643118b504b0320bcd7b5 in term 1.
I20260202 15:03:23.258729 3639 leader_election.cc:304] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17ce29b56d643118b504b0320bcd7b5, ca08a4a9ecf6452dbb02513b5c21bdc3; no voters:
I20260202 15:03:23.258898 3777 raft_consensus.cc:2804] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:23.258985 3434 raft_consensus.cc:2468] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c17ce29b56d643118b504b0320bcd7b5 in term 1.
I20260202 15:03:23.259119 3777 raft_consensus.cc:697] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [term 1 LEADER]: Becoming Leader. State: Replica: c17ce29b56d643118b504b0320bcd7b5, State: Running, Role: LEADER
I20260202 15:03:23.259445 3777 consensus_queue.cc:237] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } }
I20260202 15:03:23.262444 3285 catalog_manager.cc:5654] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c17ce29b56d643118b504b0320bcd7b5 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "c17ce29b56d643118b504b0320bcd7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca08a4a9ecf6452dbb02513b5c21bdc3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 42635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2e1df8e4d28440099808bc74a908af6" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17ce29b56d643118b504b0320bcd7b5" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 36603 } health_report { overall_health: HEALTHY } } }
W20260202 15:03:23.271162 3753 tablet.cc:2378] T 2ce6b185ca1d47548cc87040fcd12465 P c17ce29b56d643118b504b0320bcd7b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260202 15:03:23.279970 3617 tablet.cc:2378] T 2ce6b185ca1d47548cc87040fcd12465 P ca08a4a9ecf6452dbb02513b5c21bdc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260202 15:03:23.290441 3481 tablet.cc:2378] T 2ce6b185ca1d47548cc87040fcd12465 P f2e1df8e4d28440099808bc74a908af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:23.312805 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.304900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.2.164.65:41645)
0202 15:03:23.305101 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.305106 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.305204 (+ 98us) server_negotiation.cc:408] Connection header received
0202 15:03:23.305424 (+ 220us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.305428 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.305482 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.305573 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:23.306372 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.306865 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.307512 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.307730 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.309883 (+ 2153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:23.309907 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:23.309920 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:23.309945 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:23.311472 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.311846 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.311850 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.311852 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.311904 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.312089 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.312091 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.312092 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.312258 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:23.312414 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.312554 (+ 140us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.312680 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:23.338269 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.330592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.2.164.65:41645)
0202 15:03:23.330741 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.330744 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.330882 (+ 138us) server_negotiation.cc:408] Connection header received
0202 15:03:23.331014 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.331017 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.331063 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.331134 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:23.331932 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.332418 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.333144 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.333281 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.335462 (+ 2181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:23.335483 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:23.335485 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:23.335516 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:23.336795 (+ 1279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.337253 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.337256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.337257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.337297 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.337520 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.337522 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.337523 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.337659 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:23.337749 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.338042 (+ 293us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.338137 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:23.363204 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:23.355048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.2.164.65:41645)
0202 15:03:23.355231 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:23.355237 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:23.355343 (+ 106us) server_negotiation.cc:408] Connection header received
0202 15:03:23.355580 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:23.355584 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:23.355656 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:23.355754 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:23.356612 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.357223 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:23.357895 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:23.358092 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:23.360349 (+ 2257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:23.360373 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:23.360376 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:23.360403 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:23.361905 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.362297 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.362300 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.362301 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.362344 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:23.362552 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:23.362554 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:23.362555 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:23.362701 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:23.362799 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:23.362982 (+ 183us) server_negotiation.cc:300] Negotiation successful
0202 15:03:23.363075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260202 15:03:23.364717 3394 tablet_service.cc:3012] Rejecting scan request for tablet 2ce6b185ca1d47548cc87040fcd12465: Uninitialized: clean time has not yet been initialized
I20260202 15:03:23.366568 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3351
I20260202 15:03:23.372816 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3487
I20260202 15:03:23.379112 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3623
I20260202 15:03:23.386880 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3255
2026-02-02T15:03:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3494 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3793](info): set up 2 sockets
Feb 02 15:03:23 dist-test-slave-6zzp krb5kdc[3793](info): commencing operation
krb5kdc: starting...
W20260202 15:03:25.401298 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.990s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:25 dist-test-slave-6zzp krb5kdc[3793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:25.553390 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34831
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:25.664074 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:25.664301 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:25.664354 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:25.667622 3809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:25.667690 3809 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260202 15:03:25.667709 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:25.667732 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:25.667749 3809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:25.667784 3809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:25.672158 3809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:34831
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:25.673213 3809 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:25.674019 3809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:25.679729 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:25.679900 3809 server_base.cc:1060] running on GCE node
W20260202 15:03:25.679795 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:25.679769 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:25.680534 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:25.681519 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:25.682673 3809 hybrid_clock.cc:648] HybridClock initialized: now 1770044605682647 us; error 41 us; skew 500 ppm
Feb 02 15:03:25 dist-test-slave-6zzp krb5kdc[3793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:25.685254 3809 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:25.686335 3809 webserver.cc:492] Webserver started at http://127.2.164.126:40077/ using document root <none> and password file <none>
I20260202 15:03:25.686913 3809 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:25.686995 3809 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:25.687199 3809 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:25.688933 3809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "c7e0af60f9774e0d9875382e5294dca1"
format_stamp: "Formatted at 2026-02-02 15:03:25 on dist-test-slave-6zzp"
server_key: "f2a894dfa9733a47fb643e03dc27be00"
server_key_iv: "d3d6fbcaeee23e7a93c41ef23eb8a1f3"
server_key_version: "encryptionkey@0"
I20260202 15:03:25.689520 3809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "c7e0af60f9774e0d9875382e5294dca1"
format_stamp: "Formatted at 2026-02-02 15:03:25 on dist-test-slave-6zzp"
server_key: "f2a894dfa9733a47fb643e03dc27be00"
server_key_iv: "d3d6fbcaeee23e7a93c41ef23eb8a1f3"
server_key_version: "encryptionkey@0"
I20260202 15:03:25.693262 3809 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:03:25.695701 3824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:25.696853 3809 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:25.697068 3809 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "c7e0af60f9774e0d9875382e5294dca1"
format_stamp: "Formatted at 2026-02-02 15:03:25 on dist-test-slave-6zzp"
server_key: "f2a894dfa9733a47fb643e03dc27be00"
server_key_iv: "d3d6fbcaeee23e7a93c41ef23eb8a1f3"
server_key_version: "encryptionkey@0"
I20260202 15:03:25.697193 3809 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:25.714074 3809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:25.716909 3809 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:25.717133 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:25.725554 3809 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:34831
I20260202 15:03:25.725940 3886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:34831 every 8 connection(s)
I20260202 15:03:25.726598 3809 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:25.729564 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:25.729749 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3809
I20260202 15:03:25.729849 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:25.730095 2705 external_mini_cluster.cc:1468] Setting key d882bef58359106dd14e1429f60d942a
I20260202 15:03:25.735738 3887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: Bootstrap starting.
Feb 02 15:03:25 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:25.738327 3887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:25.739015 3887 log.cc:826] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:25.741256 3887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: No bootstrap required, opened a new log
I20260202 15:03:25.743275 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:25.731362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.2.164.126:34831)
0202 15:03:25.731774 (+ 412us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:25.731783 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:25.731816 (+ 33us) server_negotiation.cc:408] Connection header received
0202 15:03:25.732352 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:25.732368 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:25.732637 (+ 269us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:25.732904 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:25.733969 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:25.734739 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:25.735559 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:25.735776 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:25.738233 (+ 2457us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:25.738250 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:25.738260 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:25.738293 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:25.740394 (+ 2101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:25.740794 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:25.740799 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:25.740805 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:25.740891 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:25.741174 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:25.741177 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:25.741179 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:25.741595 (+ 416us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:25.741792 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:25.742061 (+ 269us) server_negotiation.cc:300] Negotiation successful
0202 15:03:25.742302 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260202 15:03:25.744472 3887 raft_consensus.cc:359] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e0af60f9774e0d9875382e5294dca1" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34831 } }
I20260202 15:03:25.744671 3887 raft_consensus.cc:385] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:25.744765 3887 raft_consensus.cc:740] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e0af60f9774e0d9875382e5294dca1, State: Initialized, Role: FOLLOWER
I20260202 15:03:25.745260 3887 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e0af60f9774e0d9875382e5294dca1" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34831 } }
I20260202 15:03:25.745421 3887 raft_consensus.cc:399] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:25.745509 3887 raft_consensus.cc:493] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:25.745616 3887 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:25.746407 3887 raft_consensus.cc:515] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e0af60f9774e0d9875382e5294dca1" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34831 } }
I20260202 15:03:25.746737 3887 leader_election.cc:304] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7e0af60f9774e0d9875382e5294dca1; no voters:
I20260202 15:03:25.747014 3887 leader_election.cc:290] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:25.747180 3892 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:25.747406 3892 raft_consensus.cc:697] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [term 1 LEADER]: Becoming Leader. State: Replica: c7e0af60f9774e0d9875382e5294dca1, State: Running, Role: LEADER
I20260202 15:03:25.747715 3892 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e0af60f9774e0d9875382e5294dca1" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34831 } }
I20260202 15:03:25.748154 3887 sys_catalog.cc:565] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:25.749325 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7e0af60f9774e0d9875382e5294dca1. Latest consensus state: current_term: 1 leader_uuid: "c7e0af60f9774e0d9875382e5294dca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e0af60f9774e0d9875382e5294dca1" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34831 } } }
I20260202 15:03:25.749446 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:25.749701 3893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7e0af60f9774e0d9875382e5294dca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e0af60f9774e0d9875382e5294dca1" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34831 } } }
I20260202 15:03:25.749787 3893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:25.749859 3901 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:25.752931 3901 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:25.758375 3901 catalog_manager.cc:1357] Generated new cluster ID: 6f173069557341f7a4e4af94933d6e6b
I20260202 15:03:25.758448 3901 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:25.775168 3901 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:25.776465 3901 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:25.802498 3901 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: Generated new TSK 0
I20260202 15:03:25.803337 3901 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:25.864540 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34831
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:25.972692 3915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:25.972904 3915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:25.972965 3915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:25.976262 3915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:25.976321 3915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:25.976373 3915 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:03:25.976426 3915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:25.980515 3915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:25.981571 3915 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:25.982337 3915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:25.988703 3920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:25.988716 3921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:25.988785 3923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:25.988968 3915 server_base.cc:1060] running on GCE node
I20260202 15:03:25.989398 3915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:25.989935 3915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:25.991108 3915 hybrid_clock.cc:648] HybridClock initialized: now 1770044605991090 us; error 40 us; skew 500 ppm
Feb 02 15:03:25 dist-test-slave-6zzp krb5kdc[3793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:25.993662 3915 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:25.994650 3915 webserver.cc:492] Webserver started at http://127.2.164.65:33419/ using document root <none> and password file <none>
I20260202 15:03:25.995188 3915 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:25.995260 3915 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:25.995473 3915 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:25.997658 3915 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "7156b28ce5934715b64528f5842d5927"
format_stamp: "Formatted at 2026-02-02 15:03:25 on dist-test-slave-6zzp"
server_key: "2a24c010a95fe7c798b2cdb1addb1ca3"
server_key_iv: "1dcb2c8d9654d41b6ece0f86b27a52ad"
server_key_version: "encryptionkey@0"
I20260202 15:03:25.998154 3915 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "7156b28ce5934715b64528f5842d5927"
format_stamp: "Formatted at 2026-02-02 15:03:25 on dist-test-slave-6zzp"
server_key: "2a24c010a95fe7c798b2cdb1addb1ca3"
server_key_iv: "1dcb2c8d9654d41b6ece0f86b27a52ad"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.001601 3915 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:26.003845 3930 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:26.004838 3915 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:26.004987 3915 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "7156b28ce5934715b64528f5842d5927"
format_stamp: "Formatted at 2026-02-02 15:03:25 on dist-test-slave-6zzp"
server_key: "2a24c010a95fe7c798b2cdb1addb1ca3"
server_key_iv: "1dcb2c8d9654d41b6ece0f86b27a52ad"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.005154 3915 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:26.028712 3915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:26.031880 3915 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:26.032106 3915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:26.033710 3915 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:26.033782 3915 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:26.033859 3915 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:26.033902 3915 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:26.044859 4045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:44847 every 8 connection(s)
I20260202 15:03:26.044837 3915 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:44847
I20260202 15:03:26.045962 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.034356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:35887 (local address 127.2.164.126:34831)
0202 15:03:26.034516 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.034521 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.035210 (+ 689us) server_negotiation.cc:408] Connection header received
0202 15:03:26.036053 (+ 843us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.036057 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.036114 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.036219 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.037750 (+ 1531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.038326 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.038967 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.039146 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.041917 (+ 2771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.041944 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.041948 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.041976 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.043813 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.044389 (+ 576us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.044394 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.044396 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.044450 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.044789 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.044793 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.044795 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.044952 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.045118 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.045655 (+ 537us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.045791 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260202 15:03:26.046128 3915 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:26.046475 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.034622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.2.164.65:35887)
0202 15:03:26.035057 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.035099 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.035860 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.036384 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.036398 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.036971 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.037600 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.037613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.038450 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.038459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.038849 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.038855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.039084 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.039684 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.039735 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.041752 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.043924 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.043929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.043939 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.044249 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.044554 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.044558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.044560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.044690 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.045176 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.045181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.045401 (+ 220us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.045566 (+ 165us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.045771 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":125,"threads_started":1}
I20260202 15:03:26.048038 4046 heartbeater.cc:344] Connected to a master server at 127.2.164.126:34831
I20260202 15:03:26.048206 4046 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:26.048882 4046 heartbeater.cc:507] Master 127.2.164.126:34831 requested a full tablet report, sending...
I20260202 15:03:26.050479 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 3915
I20260202 15:03:26.050483 3840 ts_manager.cc:194] Registered new tserver with Master: 7156b28ce5934715b64528f5842d5927 (127.2.164.65:44847)
I20260202 15:03:26.050604 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:26.050863 2705 external_mini_cluster.cc:1468] Setting key 000eea3a8375cdedb298e79b87f13689
I20260202 15:03:26.051975 3840 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:35887
I20260202 15:03:26.057036 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.050184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:51405 (local address 127.2.164.126:34831)
0202 15:03:26.050312 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.050316 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.050571 (+ 255us) server_negotiation.cc:408] Connection header received
0202 15:03:26.050720 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.050726 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.050774 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.050857 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.051452 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.052026 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.052707 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.052892 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.054043 (+ 1151us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.054058 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.054061 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.054086 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.055551 (+ 1465us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.055973 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.055977 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.055979 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.056028 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.056273 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.056278 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.056280 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.056421 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.056530 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.056776 (+ 246us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.056887 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260202 15:03:26.057075 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.050149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.2.164.65:51405)
0202 15:03:26.050480 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.050492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.050617 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.050871 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.050874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.051055 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.051337 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.051352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.052154 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.052157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.052584 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.052593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.052981 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.053541 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.053553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.053906 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.055700 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.055704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.055706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.055875 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.056128 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.056131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.056133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.056175 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.056521 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.056524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.056611 (+ 87us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.056722 (+ 111us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.056853 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:26.098039 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34831
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:26.205364 4057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:26.205586 4057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:26.205699 4057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:26.208906 4057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:26.208967 4057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:26.209082 4057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:03:26.209157 4057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:26.213681 4057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:26.214808 4057 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:26.215678 4057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:26.222309 4065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:26.222374 4063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:26.222324 4062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:26.222515 4057 server_base.cc:1060] running on GCE node
I20260202 15:03:26.222918 4057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:26.223420 4057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:26.224606 4057 hybrid_clock.cc:648] HybridClock initialized: now 1770044606224582 us; error 47 us; skew 500 ppm
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:26.227144 4057 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:26.228260 4057 webserver.cc:492] Webserver started at http://127.2.164.66:45081/ using document root <none> and password file <none>
I20260202 15:03:26.228843 4057 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:26.228919 4057 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:26.229169 4057 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:26.230894 4057 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "92d079bd625948cd82dc9a33cd7b607a"
format_stamp: "Formatted at 2026-02-02 15:03:26 on dist-test-slave-6zzp"
server_key: "6e5d49c09b1854becd90b59376f6c77c"
server_key_iv: "c5cd8dc116477885e9f28f5f517f4d73"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.231385 4057 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "92d079bd625948cd82dc9a33cd7b607a"
format_stamp: "Formatted at 2026-02-02 15:03:26 on dist-test-slave-6zzp"
server_key: "6e5d49c09b1854becd90b59376f6c77c"
server_key_iv: "c5cd8dc116477885e9f28f5f517f4d73"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.234795 4057 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:26.236892 4072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:26.237927 4057 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:26.238058 4057 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "92d079bd625948cd82dc9a33cd7b607a"
format_stamp: "Formatted at 2026-02-02 15:03:26 on dist-test-slave-6zzp"
server_key: "6e5d49c09b1854becd90b59376f6c77c"
server_key_iv: "c5cd8dc116477885e9f28f5f517f4d73"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.238158 4057 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:26.248392 4057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:26.251430 4057 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:26.251631 4057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:26.253067 4057 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:26.253115 4057 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:26.253186 4057 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:26.253219 4057 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:26.263058 4057 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:37205
I20260202 15:03:26.263335 4187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:37205 every 8 connection(s)
I20260202 15:03:26.264241 4057 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:26.266108 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.253778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:49667 (local address 127.2.164.126:34831)
0202 15:03:26.253933 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.253941 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.254624 (+ 683us) server_negotiation.cc:408] Connection header received
0202 15:03:26.255575 (+ 951us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.255579 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.255634 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.255734 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.257504 (+ 1770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.258326 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.259226 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.259452 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.262528 (+ 3076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.262547 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.262550 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.262581 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.264134 (+ 1553us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.264787 (+ 653us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.264791 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.264792 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.264843 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.265179 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.265182 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.265183 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.265380 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.265461 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.265896 (+ 435us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.265991 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260202 15:03:26.266687 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.254054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.2.164.66:49667)
0202 15:03:26.254475 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.254518 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.255346 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.255910 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.255928 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.256474 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.257289 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.257308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.258470 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.258474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.259093 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.259104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.259328 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.259933 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.259975 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.262386 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.264243 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.264248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.264256 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.264558 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.264941 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.264943 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.264945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.265084 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.265482 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.265487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.265678 (+ 191us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.265837 (+ 159us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.266029 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260202 15:03:26.267783 4188 heartbeater.cc:344] Connected to a master server at 127.2.164.126:34831
I20260202 15:03:26.267949 4188 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:26.268405 4188 heartbeater.cc:507] Master 127.2.164.126:34831 requested a full tablet report, sending...
I20260202 15:03:26.269402 3841 ts_manager.cc:194] Registered new tserver with Master: 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205)
I20260202 15:03:26.270113 3841 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:49667
I20260202 15:03:26.273586 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 4057
I20260202 15:03:26.273706 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:26.274019 2705 external_mini_cluster.cc:1468] Setting key 447763eab1327e94e7ba9fb95cdced56
I20260202 15:03:26.276001 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.269834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.2.164.66:57433)
0202 15:03:26.270089 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.270105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.270210 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.270464 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.270468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.270660 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.270883 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.270889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.271577 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.271583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.271952 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.271959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.272080 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.272553 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.272565 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.272836 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.274659 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.274681 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.274685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.274893 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.275126 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.275129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.275131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.275203 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.275615 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.275617 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.275695 (+ 78us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.275784 (+ 89us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.275875 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260202 15:03:26.276211 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.269903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:57433 (local address 127.2.164.126:34831)
0202 15:03:26.270035 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.270039 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.270176 (+ 137us) server_negotiation.cc:408] Connection header received
0202 15:03:26.270297 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.270300 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.270359 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.270437 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.270998 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.271463 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.272065 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.272266 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.272955 (+ 689us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.272970 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.272974 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.273046 (+ 72us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.274562 (+ 1516us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.274985 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.274988 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.274990 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.275029 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.275298 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.275300 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.275301 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.275515 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.275617 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.275869 (+ 252us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.275981 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:26.320170 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34831
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:26.424777 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:26.425055 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:26.425177 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:26.428388 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:26.428495 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:26.428597 4199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:03:26.428687 4199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:26.433175 4199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:26.434337 4199 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:26.435242 4199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:26.442230 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:26.442230 4207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:26.442302 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:26.442912 4199 server_base.cc:1060] running on GCE node
I20260202 15:03:26.443347 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:26.443966 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:26.445070 4199 hybrid_clock.cc:648] HybridClock initialized: now 1770044606445059 us; error 29 us; skew 500 ppm
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:26.447758 4199 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:26.449157 4199 webserver.cc:492] Webserver started at http://127.2.164.67:35125/ using document root <none> and password file <none>
I20260202 15:03:26.449760 4199 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:26.449813 4199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:26.450024 4199 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:26.451813 4199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "f4c12faf72ae4bfb97f67f42db406b52"
format_stamp: "Formatted at 2026-02-02 15:03:26 on dist-test-slave-6zzp"
server_key: "46a4873a978458d8f448aff1931c963c"
server_key_iv: "407319d8f4f3b84685b5aae8cc15d177"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.452301 4199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "f4c12faf72ae4bfb97f67f42db406b52"
format_stamp: "Formatted at 2026-02-02 15:03:26 on dist-test-slave-6zzp"
server_key: "46a4873a978458d8f448aff1931c963c"
server_key_iv: "407319d8f4f3b84685b5aae8cc15d177"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.455910 4199 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260202 15:03:26.458288 4214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:26.459321 4199 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:26.459457 4199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "f4c12faf72ae4bfb97f67f42db406b52"
format_stamp: "Formatted at 2026-02-02 15:03:26 on dist-test-slave-6zzp"
server_key: "46a4873a978458d8f448aff1931c963c"
server_key_iv: "407319d8f4f3b84685b5aae8cc15d177"
server_key_version: "encryptionkey@0"
I20260202 15:03:26.459565 4199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:26.469202 4199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:26.472340 4199 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:26.472565 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:26.474189 4199 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:26.474272 4199 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:26.474325 4199 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:26.474346 4199 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:26.486047 4199 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34511
I20260202 15:03:26.486060 4329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:34511 every 8 connection(s)
I20260202 15:03:26.487008 4199 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:26.487226 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.474801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:46783 (local address 127.2.164.126:34831)
0202 15:03:26.474937 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.474941 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.475987 (+ 1046us) server_negotiation.cc:408] Connection header received
0202 15:03:26.477096 (+ 1109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.477100 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.477183 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.477251 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.478765 (+ 1514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.479614 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.480620 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.480806 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.483343 (+ 2537us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.483371 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.483376 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.483412 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.485203 (+ 1791us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.485787 (+ 584us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.485791 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.485793 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.485848 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.486124 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.486130 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.486131 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.486311 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.486420 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.486934 (+ 514us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.487054 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260202 15:03:26.487995 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.475127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.2.164.67:46783)
0202 15:03:26.475836 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.475872 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.476800 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.477608 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.477616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.478017 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.478590 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.478603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.479781 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.479785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.480498 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.480509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.480768 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.481525 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.481565 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.483177 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.485342 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.485348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.485363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.485663 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.485941 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.485945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.485949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.486029 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.486418 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.486425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.486681 (+ 256us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.486897 (+ 216us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.487160 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":131,"threads_started":1}
I20260202 15:03:26.489243 4330 heartbeater.cc:344] Connected to a master server at 127.2.164.126:34831
I20260202 15:03:26.489392 4330 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:26.489969 4330 heartbeater.cc:507] Master 127.2.164.126:34831 requested a full tablet report, sending...
I20260202 15:03:26.491019 3840 ts_manager.cc:194] Registered new tserver with Master: f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
I20260202 15:03:26.491762 3840 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:46783
I20260202 15:03:26.495707 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 4199
I20260202 15:03:26.495831 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:26.496109 2705 external_mini_cluster.cc:1468] Setting key 6c8ead10bdae72f2de6285dbb936bc16
I20260202 15:03:26.497926 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.491491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:44231 (local address 127.2.164.126:34831)
0202 15:03:26.491614 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.491618 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.491821 (+ 203us) server_negotiation.cc:408] Connection header received
0202 15:03:26.491971 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.491975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.492031 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.492106 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.492670 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.493392 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.493955 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.494081 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.494801 (+ 720us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.494814 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.494817 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.494838 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.496380 (+ 1542us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.496824 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.496827 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.496828 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.496867 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.497178 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.497181 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.497183 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.497367 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.497469 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.497735 (+ 266us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.497820 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260202 15:03:26.497982 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.491415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.2.164.67:44231)
0202 15:03:26.491740 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.491752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.491878 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.492135 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.492139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.492341 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.492557 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.492563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.493496 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.493498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.493858 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.493865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.493951 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.494408 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.494420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.494683 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.496503 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.496507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.496509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.496707 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.496995 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.496999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.497020 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.497081 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.497463 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.497465 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.497573 (+ 108us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.497679 (+ 106us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.497814 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260202 15:03:26.498611 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:26.522840 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.515501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.2.164.126:34831)
0202 15:03:26.515640 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.515643 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.515655 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:03:26.515697 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.515700 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.515746 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.515814 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.516666 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.517225 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.517885 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.518018 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.520082 (+ 2064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.520102 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.520104 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.520130 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.521545 (+ 1415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.521872 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.521875 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.521876 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.521916 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.522092 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.522094 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.522095 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.522235 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.522335 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.522517 (+ 182us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.522612 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:03:26.525513 3840 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:26.527946 3840 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:26.541628 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.536748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44847 (local address 127.0.0.1:46028)
0202 15:03:26.537631 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.537645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.537746 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.538570 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.538573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.538592 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.538840 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.538846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.540384 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.540387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.541236 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.541245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.541322 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.541339 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.541382 (+ 43us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.541447 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"spinlock_wait_cycles":896,"thread_start_us":113,"threads_started":1}
I20260202 15:03:26.541633 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.536620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37205 (local address 127.0.0.1:60612)
0202 15:03:26.537219 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.537257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.537427 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.538461 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.538467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.538504 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.538838 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.538845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.539958 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.539964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.540967 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.540979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.541124 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.541192 (+ 68us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.541293 (+ 101us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.541400 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":84,"threads_started":1}
I20260202 15:03:26.542510 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.537360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34511 (local address 127.0.0.1:52302)
0202 15:03:26.537804 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.537818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.537905 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.538623 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.538626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.538640 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.538865 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.538874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.540283 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.540287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.542170 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.542180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.542278 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.542294 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.542342 (+ 48us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.542396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":82,"threads_started":1}
I20260202 15:03:26.542888 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.537154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.2.164.65:44847)
0202 15:03:26.537456 (+ 302us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.537464 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.538149 (+ 685us) server_negotiation.cc:408] Connection header received
0202 15:03:26.538232 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.538238 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.538401 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.538989 (+ 588us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.539156 (+ 167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.540252 (+ 1096us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.541894 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.542458 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.542591 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.542657 (+ 66us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.542750 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260202 15:03:26.543258 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.536916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.2.164.66:37205)
0202 15:03:26.537981 (+ 1065us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.537988 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.538010 (+ 22us) server_negotiation.cc:408] Connection header received
0202 15:03:26.538093 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.538100 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.538283 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.538996 (+ 713us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.539123 (+ 127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.539824 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.541951 (+ 2127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.542791 (+ 840us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.542909 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.542980 (+ 71us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.543083 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":473,"threads_started":1}
I20260202 15:03:26.543644 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.537784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.2.164.67:34511)
0202 15:03:26.538103 (+ 319us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.538108 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.538126 (+ 18us) server_negotiation.cc:408] Connection header received
0202 15:03:26.538190 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.538194 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.538315 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.538444 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.538993 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.540143 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.542682 (+ 2539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.543239 (+ 557us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.543342 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.543400 (+ 58us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.543489 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":117,"threads_started":1}
I20260202 15:03:26.546435 4122 tablet_service.cc:1505] Processing CreateTablet for tablet a4cbc1e014aa4698819e1524a1d82c47 (DEFAULT_TABLE table=test-table [id=fc83272f17294a669e480b568b93381b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:26.546435 4264 tablet_service.cc:1505] Processing CreateTablet for tablet a4cbc1e014aa4698819e1524a1d82c47 (DEFAULT_TABLE table=test-table [id=fc83272f17294a669e480b568b93381b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:26.546821 3980 tablet_service.cc:1505] Processing CreateTablet for tablet a4cbc1e014aa4698819e1524a1d82c47 (DEFAULT_TABLE table=test-table [id=fc83272f17294a669e480b568b93381b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:26.547489 4264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4cbc1e014aa4698819e1524a1d82c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.547729 3980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4cbc1e014aa4698819e1524a1d82c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.547809 4122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4cbc1e014aa4698819e1524a1d82c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.553653 4353 tablet_bootstrap.cc:492] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Bootstrap starting.
I20260202 15:03:26.554471 4354 tablet_bootstrap.cc:492] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Bootstrap starting.
I20260202 15:03:26.554514 4355 tablet_bootstrap.cc:492] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Bootstrap starting.
I20260202 15:03:26.556310 4355 tablet_bootstrap.cc:654] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.556524 4354 tablet_bootstrap.cc:654] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.556543 4353 tablet_bootstrap.cc:654] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.557549 4355 log.cc:826] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:26.557675 4353 log.cc:826] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:26.557668 4354 log.cc:826] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:26.559259 4354 tablet_bootstrap.cc:492] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: No bootstrap required, opened a new log
I20260202 15:03:26.559259 4355 tablet_bootstrap.cc:492] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: No bootstrap required, opened a new log
I20260202 15:03:26.559468 4354 ts_tablet_manager.cc:1403] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:26.559468 4355 ts_tablet_manager.cc:1403] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:26.560400 4353 tablet_bootstrap.cc:492] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: No bootstrap required, opened a new log
I20260202 15:03:26.560582 4353 ts_tablet_manager.cc:1403] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260202 15:03:26.562100 4354 raft_consensus.cc:359] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.562330 4354 raft_consensus.cc:385] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.562386 4354 raft_consensus.cc:740] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c12faf72ae4bfb97f67f42db406b52, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.562579 4355 raft_consensus.cc:359] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.562810 4355 raft_consensus.cc:385] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.562865 4355 raft_consensus.cc:740] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d079bd625948cd82dc9a33cd7b607a, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.562868 4354 consensus_queue.cc:260] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.563385 4355 consensus_queue.cc:260] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.563768 4330 heartbeater.cc:499] Master 127.2.164.126:34831 was elected leader, sending a full tablet report...
I20260202 15:03:26.563773 4354 ts_tablet_manager.cc:1434] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:03:26.564119 4353 raft_consensus.cc:359] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.564306 4353 raft_consensus.cc:385] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.564364 4353 raft_consensus.cc:740] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7156b28ce5934715b64528f5842d5927, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.564726 4353 consensus_queue.cc:260] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.565485 4188 heartbeater.cc:499] Master 127.2.164.126:34831 was elected leader, sending a full tablet report...
I20260202 15:03:26.565765 4353 ts_tablet_manager.cc:1434] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260202 15:03:26.566124 4046 heartbeater.cc:499] Master 127.2.164.126:34831 was elected leader, sending a full tablet report...
I20260202 15:03:26.567441 4355 ts_tablet_manager.cc:1434] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260202 15:03:26.733773 4360 raft_consensus.cc:493] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:26.733986 4360 raft_consensus.cc:515] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.735266 4360 leader_election.cc:290] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7156b28ce5934715b64528f5842d5927 (127.2.164.65:44847), f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
W20260202 15:03:26.737955 4331 tablet.cc:2378] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:03:26.738759 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.735581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34511 (local address 127.2.164.66:56207)
0202 15:03:26.735792 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.735809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.735911 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.736396 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.736399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.736416 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.736685 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.736693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.737568 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.737574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.738338 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.738348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.738462 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.738489 (+ 27us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.738543 (+ 54us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.738604 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260202 15:03:26.739290 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.735581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44847 (local address 127.2.164.66:38825)
0202 15:03:26.736091 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.736114 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.736193 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.736623 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.736626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.736641 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.736925 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.736933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.737959 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.737962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.738928 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.738938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.739085 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.739107 (+ 22us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.739148 (+ 41us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.739191 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":56,"thread_start_us":152,"threads_started":1}
I20260202 15:03:26.739290 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.735852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:56207 (local address 127.2.164.67:34511)
0202 15:03:26.736076 (+ 224us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.736081 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.736095 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:03:26.736145 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.736148 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.736201 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.736286 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.736827 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.737454 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.738512 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.739004 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.739077 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.739124 (+ 47us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.739184 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260202 15:03:26.739815 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.736000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:38825 (local address 127.2.164.65:44847)
0202 15:03:26.736149 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.736154 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.736167 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:03:26.736322 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.736327 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.736376 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.736490 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.737079 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.737835 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.739101 (+ 1266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.739575 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.739615 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.739675 (+ 60us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.739729 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:03:26.739912 4284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4cbc1e014aa4698819e1524a1d82c47" candidate_uuid: "92d079bd625948cd82dc9a33cd7b607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c12faf72ae4bfb97f67f42db406b52" is_pre_election: true
I20260202 15:03:26.740202 4284 raft_consensus.cc:2468] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d079bd625948cd82dc9a33cd7b607a in term 0.
I20260202 15:03:26.740391 4000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4cbc1e014aa4698819e1524a1d82c47" candidate_uuid: "92d079bd625948cd82dc9a33cd7b607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7156b28ce5934715b64528f5842d5927" is_pre_election: true
I20260202 15:03:26.740679 4000 raft_consensus.cc:2468] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d079bd625948cd82dc9a33cd7b607a in term 0.
I20260202 15:03:26.740694 4074 leader_election.cc:304] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d079bd625948cd82dc9a33cd7b607a, f4c12faf72ae4bfb97f67f42db406b52; no voters:
I20260202 15:03:26.740967 4360 raft_consensus.cc:2804] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:26.741072 4360 raft_consensus.cc:493] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:26.741106 4360 raft_consensus.cc:3060] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:26.742187 4360 raft_consensus.cc:515] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.742574 4360 leader_election.cc:290] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [CANDIDATE]: Term 1 election: Requested vote from peers 7156b28ce5934715b64528f5842d5927 (127.2.164.65:44847), f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
I20260202 15:03:26.742962 4000 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4cbc1e014aa4698819e1524a1d82c47" candidate_uuid: "92d079bd625948cd82dc9a33cd7b607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7156b28ce5934715b64528f5842d5927"
I20260202 15:03:26.742964 4284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4cbc1e014aa4698819e1524a1d82c47" candidate_uuid: "92d079bd625948cd82dc9a33cd7b607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c12faf72ae4bfb97f67f42db406b52"
I20260202 15:03:26.743094 4000 raft_consensus.cc:3060] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:26.743110 4284 raft_consensus.cc:3060] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:26.744059 4000 raft_consensus.cc:2468] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d079bd625948cd82dc9a33cd7b607a in term 1.
I20260202 15:03:26.744091 4284 raft_consensus.cc:2468] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d079bd625948cd82dc9a33cd7b607a in term 1.
I20260202 15:03:26.744400 4073 leader_election.cc:304] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7156b28ce5934715b64528f5842d5927, 92d079bd625948cd82dc9a33cd7b607a; no voters:
I20260202 15:03:26.744573 4360 raft_consensus.cc:2804] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:26.744788 4360 raft_consensus.cc:697] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 1 LEADER]: Becoming Leader. State: Replica: 92d079bd625948cd82dc9a33cd7b607a, State: Running, Role: LEADER
I20260202 15:03:26.745182 4360 consensus_queue.cc:237] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.748396 3840 catalog_manager.cc:5654] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a reported cstate change: term changed from 0 to 1, leader changed from <none> to 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66). New cstate: current_term: 1 leader_uuid: "92d079bd625948cd82dc9a33cd7b607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } health_report { overall_health: UNKNOWN } } }
W20260202 15:03:26.765300 4189 tablet.cc:2378] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:03:26.783756 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.780907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.2.164.66:37205)
0202 15:03:26.781090 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.781094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.781134 (+ 40us) server_negotiation.cc:408] Connection header received
0202 15:03:26.781324 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.781326 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.781382 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.781470 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:26.781795 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.782254 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.782846 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.782976 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.783022 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:26.783433 (+ 411us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:26.783536 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.783610 (+ 74us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.783653 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:03:26.788754 4000 raft_consensus.cc:1275] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Refusing update from remote peer 92d079bd625948cd82dc9a33cd7b607a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:26.788817 4284 raft_consensus.cc:1275] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Refusing update from remote peer 92d079bd625948cd82dc9a33cd7b607a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:26.789465 4363 consensus_queue.cc:1048] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:26.789634 4360 consensus_queue.cc:1048] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:26.796355 4367 mvcc.cc:204] Tried to move back new op lower bound from 7250102709400346624 to 7250102709230985216. Current Snapshot: MvccSnapshot[applied={T|T < 7250102709400346624}]
I20260202 15:03:26.806381 3840 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:26.806559 3840 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:26.808831 3840 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: Sending DeleteTablet for 3 replicas of tablet a4cbc1e014aa4698819e1524a1d82c47
I20260202 15:03:26.809664 4122 tablet_service.cc:1552] Processing DeleteTablet for tablet a4cbc1e014aa4698819e1524a1d82c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:26 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60612
I20260202 15:03:26.809669 3980 tablet_service.cc:1552] Processing DeleteTablet for tablet a4cbc1e014aa4698819e1524a1d82c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:26 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:46028
I20260202 15:03:26.809851 4264 tablet_service.cc:1552] Processing DeleteTablet for tablet a4cbc1e014aa4698819e1524a1d82c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:26 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:52302
I20260202 15:03:26.810163 4378 tablet_replica.cc:333] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: stopping tablet replica
I20260202 15:03:26.810285 4377 tablet_replica.cc:333] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: stopping tablet replica
I20260202 15:03:26.810526 4377 raft_consensus.cc:2243] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:26.810532 4378 raft_consensus.cc:2243] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:26.810734 3840 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260202 15:03:26.810880 4377 raft_consensus.cc:2272] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:26.811156 4378 raft_consensus.cc:2272] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260202 15:03:26.811156 3840 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:26.812232 4377 ts_tablet_manager.cc:1916] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:26.812542 4378 ts_tablet_manager.cc:1916] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:26.814903 4377 ts_tablet_manager.cc:1929] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:03:26.814996 4377 log.cc:1199] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/a4cbc1e014aa4698819e1524a1d82c47
I20260202 15:03:26.815323 4377 ts_tablet_manager.cc:1950] T a4cbc1e014aa4698819e1524a1d82c47 P 92d079bd625948cd82dc9a33cd7b607a: Deleting consensus metadata
I20260202 15:03:26.815426 4378 ts_tablet_manager.cc:1929] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:03:26.815568 4378 log.cc:1199] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/a4cbc1e014aa4698819e1524a1d82c47
I20260202 15:03:26.815945 4378 ts_tablet_manager.cc:1950] T a4cbc1e014aa4698819e1524a1d82c47 P 7156b28ce5934715b64528f5842d5927: Deleting consensus metadata
I20260202 15:03:26.816663 3980 tablet_service.cc:1505] Processing CreateTablet for tablet 208477fd94c84c2bb5f17499fa1c1890 (DEFAULT_TABLE table=test-table [id=7cc35a02dfbb4dcb9f33c38e5437e1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:26.816707 4379 tablet_replica.cc:333] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: stopping tablet replica
I20260202 15:03:26.816915 3980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 208477fd94c84c2bb5f17499fa1c1890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.816960 4379 raft_consensus.cc:2243] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:26.817289 4379 raft_consensus.cc:2272] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:26.817843 3826 catalog_manager.cc:4985] TS 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205): tablet a4cbc1e014aa4698819e1524a1d82c47 (table test-table [id=fc83272f17294a669e480b568b93381b]) successfully deleted
I20260202 15:03:26.818259 4264 tablet_service.cc:1505] Processing CreateTablet for tablet 208477fd94c84c2bb5f17499fa1c1890 (DEFAULT_TABLE table=test-table [id=7cc35a02dfbb4dcb9f33c38e5437e1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:26.818490 4264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 208477fd94c84c2bb5f17499fa1c1890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.818996 4353 tablet_bootstrap.cc:492] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Bootstrap starting.
I20260202 15:03:26.819449 4379 ts_tablet_manager.cc:1916] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:26.820052 4353 tablet_bootstrap.cc:654] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.820724 3825 catalog_manager.cc:4985] TS 7156b28ce5934715b64528f5842d5927 (127.2.164.65:44847): tablet a4cbc1e014aa4698819e1524a1d82c47 (table test-table [id=fc83272f17294a669e480b568b93381b]) successfully deleted
I20260202 15:03:26.820824 4354 tablet_bootstrap.cc:492] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Bootstrap starting.
I20260202 15:03:26.821345 4122 tablet_service.cc:1505] Processing CreateTablet for tablet 208477fd94c84c2bb5f17499fa1c1890 (DEFAULT_TABLE table=test-table [id=7cc35a02dfbb4dcb9f33c38e5437e1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:26.821614 4122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 208477fd94c84c2bb5f17499fa1c1890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.821822 4354 tablet_bootstrap.cc:654] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.821990 4379 ts_tablet_manager.cc:1929] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:03:26.822091 4379 log.cc:1199] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/a4cbc1e014aa4698819e1524a1d82c47
I20260202 15:03:26.822391 4379 ts_tablet_manager.cc:1950] T a4cbc1e014aa4698819e1524a1d82c47 P f4c12faf72ae4bfb97f67f42db406b52: Deleting consensus metadata
I20260202 15:03:26.823565 4354 tablet_bootstrap.cc:492] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: No bootstrap required, opened a new log
I20260202 15:03:26.823660 4354 ts_tablet_manager.cc:1403] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:03:26.824012 4355 tablet_bootstrap.cc:492] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Bootstrap starting.
I20260202 15:03:26.824155 4354 raft_consensus.cc:359] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.824254 4354 raft_consensus.cc:385] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.824285 4354 raft_consensus.cc:740] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c12faf72ae4bfb97f67f42db406b52, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.824388 4354 consensus_queue.cc:260] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.824656 4354 ts_tablet_manager.cc:1434] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:26.824945 3826 catalog_manager.cc:4985] TS f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511): tablet a4cbc1e014aa4698819e1524a1d82c47 (table test-table [id=fc83272f17294a669e480b568b93381b]) successfully deleted
I20260202 15:03:26.825119 4355 tablet_bootstrap.cc:654] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.826063 4353 tablet_bootstrap.cc:492] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: No bootstrap required, opened a new log
I20260202 15:03:26.826143 4353 ts_tablet_manager.cc:1403] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260202 15:03:26.826704 4353 raft_consensus.cc:359] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.826825 4353 raft_consensus.cc:385] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.826865 4353 raft_consensus.cc:740] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7156b28ce5934715b64528f5842d5927, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.827001 4353 consensus_queue.cc:260] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.827294 4353 ts_tablet_manager.cc:1434] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260202 15:03:26.829864 4355 tablet_bootstrap.cc:492] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: No bootstrap required, opened a new log
I20260202 15:03:26.829959 4355 ts_tablet_manager.cc:1403] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260202 15:03:26.830467 4355 raft_consensus.cc:359] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.830613 4355 raft_consensus.cc:385] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.830660 4355 raft_consensus.cc:740] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d079bd625948cd82dc9a33cd7b607a, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.830837 4355 consensus_queue.cc:260] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.831113 4355 ts_tablet_manager.cc:1434] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260202 15:03:26.882254 4361 raft_consensus.cc:493] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:26.882411 4361 raft_consensus.cc:515] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.883648 4361 leader_election.cc:290] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205), f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
I20260202 15:03:26.887410 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.883932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37205 (local address 127.2.164.65:47659)
0202 15:03:26.884307 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.884322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.884484 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.884898 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.884900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.884915 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.885268 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.885278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.886310 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.886313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.886974 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.886982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.887112 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.887135 (+ 23us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.887187 (+ 52us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.887244 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260202 15:03:26.887861 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.883984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:47659 (local address 127.2.164.66:37205)
0202 15:03:26.884141 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.884146 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.884383 (+ 237us) server_negotiation.cc:408] Connection header received
0202 15:03:26.884625 (+ 242us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.884629 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.884685 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.884742 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.885409 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.886187 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.887099 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.887588 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.887634 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.887687 (+ 53us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.887760 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260202 15:03:26.888372 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.884347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34511 (local address 127.2.164.65:46141)
0202 15:03:26.884661 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.884690 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.884779 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.885053 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.885056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.885069 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:26.885715 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.885723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.886487 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.886491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.887416 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.887426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.888129 (+ 703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.888146 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.888228 (+ 82us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.888283 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
I20260202 15:03:26.888827 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.884454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:46141 (local address 127.2.164.67:34511)
0202 15:03:26.884573 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.884578 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.884772 (+ 194us) server_negotiation.cc:408] Connection header received
0202 15:03:26.884864 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.884867 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.884922 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.885036 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:26.885844 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.886360 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.887569 (+ 1209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.888048 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.888085 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.888677 (+ 592us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.888742 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:03:26.888947 4136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "208477fd94c84c2bb5f17499fa1c1890" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d079bd625948cd82dc9a33cd7b607a" is_pre_election: true
I20260202 15:03:26.889214 4136 raft_consensus.cc:2468] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 0.
I20260202 15:03:26.889334 4284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "208477fd94c84c2bb5f17499fa1c1890" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c12faf72ae4bfb97f67f42db406b52" is_pre_election: true
I20260202 15:03:26.889487 4284 raft_consensus.cc:2468] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 0.
I20260202 15:03:26.889762 3932 leader_election.cc:304] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7156b28ce5934715b64528f5842d5927, 92d079bd625948cd82dc9a33cd7b607a; no voters:
I20260202 15:03:26.890038 4361 raft_consensus.cc:2804] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:26.890097 4361 raft_consensus.cc:493] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:26.890134 4361 raft_consensus.cc:3060] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:26.891259 4361 raft_consensus.cc:515] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.891624 4361 leader_election.cc:290] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 election: Requested vote from peers 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205), f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
I20260202 15:03:26.891968 4136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "208477fd94c84c2bb5f17499fa1c1890" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d079bd625948cd82dc9a33cd7b607a"
I20260202 15:03:26.892011 4284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "208477fd94c84c2bb5f17499fa1c1890" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c12faf72ae4bfb97f67f42db406b52"
I20260202 15:03:26.892084 4136 raft_consensus.cc:3060] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:26.892133 4284 raft_consensus.cc:3060] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:26.892930 4284 raft_consensus.cc:2468] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 1.
I20260202 15:03:26.893208 4136 raft_consensus.cc:2468] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 1.
I20260202 15:03:26.893241 3932 leader_election.cc:304] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7156b28ce5934715b64528f5842d5927, f4c12faf72ae4bfb97f67f42db406b52; no voters:
I20260202 15:03:26.893428 4361 raft_consensus.cc:2804] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:26.893687 4361 raft_consensus.cc:697] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 1 LEADER]: Becoming Leader. State: Replica: 7156b28ce5934715b64528f5842d5927, State: Running, Role: LEADER
I20260202 15:03:26.893939 4361 consensus_queue.cc:237] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.896147 3841 catalog_manager.cc:5654] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7156b28ce5934715b64528f5842d5927 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } health_report { overall_health: UNKNOWN } } }
Feb 02 15:03:26 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:26.948824 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.939121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.2.164.126:34831)
0202 15:03:26.939274 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:26.939279 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:26.939496 (+ 217us) server_negotiation.cc:408] Connection header received
0202 15:03:26.939601 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:26.939605 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:26.939656 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:26.939778 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:26.940533 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.941370 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.942076 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.942246 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.944677 (+ 2431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:26.944700 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:26.944704 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:26.944733 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:26.946836 (+ 2103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.947312 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.947316 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.947318 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.947369 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:26.947756 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:26.947761 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:26.947764 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:26.947963 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:26.948063 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:26.948392 (+ 329us) server_negotiation.cc:300] Negotiation successful
0202 15:03:26.948537 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:03:26.949175 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:26.939066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34831 (local address 127.0.0.1:55956)
0202 15:03:26.939381 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:26.939410 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:26.939523 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:26.939793 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:26.939795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:26.940183 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:26.940372 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.940378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.941516 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:26.941520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:26.941906 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:26.941913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:26.942100 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:26.942852 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:26.942866 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:26.944539 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:26.946960 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.946967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.946971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.947211 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.947499 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:26.947502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:26.947503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:26.947566 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:26.948077 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:26.948084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:26.948191 (+ 107us) client_negotiation.cc:769] Sending connection context
0202 15:03:26.948290 (+ 99us) client_negotiation.cc:240] Negotiation successful
0202 15:03:26.948661 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":109,"thread_start_us":149,"threads_started":1}
I20260202 15:03:26.953833 3841 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260202 15:03:26.954588 3841 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:26.959812 3980 tablet_service.cc:1505] Processing CreateTablet for tablet 22f3be75371642afae1b84abbdc6644a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93256e8d1b17410997dbecb1dc07daf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:03:26.960253 3980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f3be75371642afae1b84abbdc6644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.960250 4264 tablet_service.cc:1505] Processing CreateTablet for tablet 22f3be75371642afae1b84abbdc6644a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93256e8d1b17410997dbecb1dc07daf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:03:26.960637 4264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f3be75371642afae1b84abbdc6644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.963151 4353 tablet_bootstrap.cc:492] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: Bootstrap starting.
I20260202 15:03:26.963425 4354 tablet_bootstrap.cc:492] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: Bootstrap starting.
I20260202 15:03:26.963727 4122 tablet_service.cc:1505] Processing CreateTablet for tablet 22f3be75371642afae1b84abbdc6644a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93256e8d1b17410997dbecb1dc07daf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:03:26.964052 4122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f3be75371642afae1b84abbdc6644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:26.964238 4353 tablet_bootstrap.cc:654] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.964465 4354 tablet_bootstrap.cc:654] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.965756 4353 tablet_bootstrap.cc:492] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: No bootstrap required, opened a new log
I20260202 15:03:26.965844 4353 ts_tablet_manager.cc:1403] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260202 15:03:26.966243 4354 tablet_bootstrap.cc:492] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: No bootstrap required, opened a new log
I20260202 15:03:26.966323 4354 ts_tablet_manager.cc:1403] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:03:26.966356 4353 raft_consensus.cc:359] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.966485 4353 raft_consensus.cc:385] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.966516 4353 raft_consensus.cc:740] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7156b28ce5934715b64528f5842d5927, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.966642 4353 consensus_queue.cc:260] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.966805 4354 raft_consensus.cc:359] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.966907 4354 raft_consensus.cc:385] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.966939 4354 raft_consensus.cc:740] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c12faf72ae4bfb97f67f42db406b52, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.966952 4353 ts_tablet_manager.cc:1434] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:26.966974 4361 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:26.967077 4361 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:26.967065 4354 consensus_queue.cc:260] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.967321 4354 ts_tablet_manager.cc:1434] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:26.967579 4359 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:26.967700 4359 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:26.968655 4355 tablet_bootstrap.cc:492] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: Bootstrap starting.
I20260202 15:03:26.969668 4355 tablet_bootstrap.cc:654] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:26.971220 4355 tablet_bootstrap.cc:492] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: No bootstrap required, opened a new log
I20260202 15:03:26.971298 4355 ts_tablet_manager.cc:1403] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:03:26.971769 4355 raft_consensus.cc:359] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.971863 4355 raft_consensus.cc:385] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:26.971885 4355 raft_consensus.cc:740] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d079bd625948cd82dc9a33cd7b607a, State: Initialized, Role: FOLLOWER
I20260202 15:03:26.972034 4355 consensus_queue.cc:260] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:26.972247 4363 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:26.972298 4355 ts_tablet_manager.cc:1434] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260202 15:03:26.972371 4363 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260202 15:03:27.048475 4047 tablet.cc:2378] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:03:27.093595 4361 raft_consensus.cc:493] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:27.093732 4361 raft_consensus.cc:515] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:27.094163 4361 leader_election.cc:290] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205), f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
I20260202 15:03:27.094616 4136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f3be75371642afae1b84abbdc6644a" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d079bd625948cd82dc9a33cd7b607a" is_pre_election: true
I20260202 15:03:27.094671 4284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f3be75371642afae1b84abbdc6644a" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c12faf72ae4bfb97f67f42db406b52" is_pre_election: true
I20260202 15:03:27.094774 4136 raft_consensus.cc:2468] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 0.
I20260202 15:03:27.094852 4284 raft_consensus.cc:2468] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 0.
I20260202 15:03:27.095105 3932 leader_election.cc:304] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7156b28ce5934715b64528f5842d5927, 92d079bd625948cd82dc9a33cd7b607a; no voters:
I20260202 15:03:27.095284 4361 raft_consensus.cc:2804] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:27.095343 4361 raft_consensus.cc:493] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:27.095382 4361 raft_consensus.cc:3060] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:27.096249 4361 raft_consensus.cc:515] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:27.096592 4361 leader_election.cc:290] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 election: Requested vote from peers 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205), f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511)
I20260202 15:03:27.096939 4136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f3be75371642afae1b84abbdc6644a" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d079bd625948cd82dc9a33cd7b607a"
I20260202 15:03:27.097072 4136 raft_consensus.cc:3060] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:27.097040 4284 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f3be75371642afae1b84abbdc6644a" candidate_uuid: "7156b28ce5934715b64528f5842d5927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c12faf72ae4bfb97f67f42db406b52"
I20260202 15:03:27.097172 4284 raft_consensus.cc:3060] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:27.097908 4136 raft_consensus.cc:2468] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 1.
I20260202 15:03:27.098083 4284 raft_consensus.cc:2468] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7156b28ce5934715b64528f5842d5927 in term 1.
I20260202 15:03:27.098229 3932 leader_election.cc:304] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7156b28ce5934715b64528f5842d5927, 92d079bd625948cd82dc9a33cd7b607a; no voters:
I20260202 15:03:27.098397 4361 raft_consensus.cc:2804] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:27.098485 4361 raft_consensus.cc:697] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [term 1 LEADER]: Becoming Leader. State: Replica: 7156b28ce5934715b64528f5842d5927, State: Running, Role: LEADER
I20260202 15:03:27.098631 4361 consensus_queue.cc:237] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } }
I20260202 15:03:27.099351 4385 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: TxnStatusTablet state changed. Reason: New leader 7156b28ce5934715b64528f5842d5927. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.099442 4385 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:03:27.099725 4397 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260202 15:03:27.099867 4397 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260202 15:03:27.100440 3841 catalog_manager.cc:5654] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7156b28ce5934715b64528f5842d5927 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } health_report { overall_health: UNKNOWN } } }
I20260202 15:03:27.127965 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.124694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.2.164.65:44847)
0202 15:03:27.124841 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:27.124846 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:27.124862 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:03:27.125059 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:27.125062 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:27.125120 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:27.125220 (+ 100us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:27.125706 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.126181 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.126856 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.127052 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.127103 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:27.127552 (+ 449us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:27.127678 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:27.127771 (+ 93us) server_negotiation.cc:300] Negotiation successful
0202 15:03:27.127822 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260202 15:03:27.127965 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.124574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44847 (local address 127.0.0.1:46038)
0202 15:03:27.124714 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:27.124733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:27.124862 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:27.125340 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:27.125344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:27.125355 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:03:27.125573 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:27.125580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.126299 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.126304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:27.126728 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:27.126734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.126870 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.126919 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:03:27.127654 (+ 735us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:03:27.127672 (+ 18us) client_negotiation.cc:769] Sending connection context
0202 15:03:27.127755 (+ 83us) client_negotiation.cc:240] Negotiation successful
0202 15:03:27.127837 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260202 15:03:27.130954 4284 raft_consensus.cc:1275] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Refusing update from remote peer 7156b28ce5934715b64528f5842d5927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:27.131152 4136 raft_consensus.cc:1275] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Refusing update from remote peer 7156b28ce5934715b64528f5842d5927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:27.131521 4385 consensus_queue.cc:1048] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:27.131683 4361 consensus_queue.cc:1048] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:27.133530 4363 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: TxnStatusTablet state changed. Reason: New leader 7156b28ce5934715b64528f5842d5927. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.133567 4359 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: TxnStatusTablet state changed. Reason: New leader 7156b28ce5934715b64528f5842d5927. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.133646 4363 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:27.133671 4359 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:27.136751 4402 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.136893 4402 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:03:27.137075 4402 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.137171 4402 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 7156b28ce5934715b64528f5842d5927: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:03:27.137113 4359 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.137207 4359 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P f4c12faf72ae4bfb97f67f42db406b52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:27.137827 4368 mvcc.cc:204] Tried to move back new op lower bound from 7250102710803042304 to 7250102710677557248. Current Snapshot: MvccSnapshot[applied={T|T < 7250102710803042304}]
I20260202 15:03:27.136945 4366 mvcc.cc:204] Tried to move back new op lower bound from 7250102710803042304 to 7250102710677557248. Current Snapshot: MvccSnapshot[applied={T|T < 7250102710803042304}]
I20260202 15:03:27.139683 4363 tablet_replica.cc:442] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7156b28ce5934715b64528f5842d5927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7156b28ce5934715b64528f5842d5927" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44847 } } peers { permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 } } peers { permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 } } }
I20260202 15:03:27.139782 4363 tablet_replica.cc:445] T 22f3be75371642afae1b84abbdc6644a P 92d079bd625948cd82dc9a33cd7b607a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:27.147135 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.144314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.2.164.65:44847)
0202 15:03:27.144439 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:27.144443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:27.144459 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:03:27.144580 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:27.144582 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:27.144628 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:27.144713 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:27.145149 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.145651 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.146493 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.146624 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.146663 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:27.146727 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:27.146798 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:27.146924 (+ 126us) server_negotiation.cc:300] Negotiation successful
0202 15:03:27.146969 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:03:27.157642 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.154608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:54711 (local address 127.2.164.65:44847)
0202 15:03:27.154742 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:27.154746 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:27.154956 (+ 210us) server_negotiation.cc:408] Connection header received
0202 15:03:27.155066 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:27.155070 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:27.155122 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:27.155202 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:27.155547 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.156283 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.156924 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.157132 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.157178 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:27.157253 (+ 75us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:27.157332 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:27.157436 (+ 104us) server_negotiation.cc:300] Negotiation successful
0202 15:03:27.157483 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260202 15:03:27.157650 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.154505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44847 (local address 127.2.164.65:54711)
0202 15:03:27.154869 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:27.154881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:27.154979 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:27.155217 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:27.155219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:27.155225 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:03:27.155444 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:27.155448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.156392 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.156395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:27.156823 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:27.156830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.156917 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.156954 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:03:27.157343 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:03:27.157347 (+ 4us) client_negotiation.cc:769] Sending connection context
0202 15:03:27.157403 (+ 56us) client_negotiation.cc:240] Negotiation successful
0202 15:03:27.157466 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260202 15:03:27.164188 4284 raft_consensus.cc:1275] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Refusing update from remote peer 7156b28ce5934715b64528f5842d5927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:27.164302 4136 raft_consensus.cc:1275] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Refusing update from remote peer 7156b28ce5934715b64528f5842d5927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:27.164641 4385 consensus_queue.cc:1048] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c12faf72ae4bfb97f67f42db406b52" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:27.164788 4361 consensus_queue.cc:1048] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d079bd625948cd82dc9a33cd7b607a" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:27.208082 3840 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:27.208247 3840 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:27.210321 3840 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c7e0af60f9774e0d9875382e5294dca1: Sending DeleteTablet for 3 replicas of tablet 208477fd94c84c2bb5f17499fa1c1890
I20260202 15:03:27.210879 3980 tablet_service.cc:1552] Processing DeleteTablet for tablet 208477fd94c84c2bb5f17499fa1c1890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:27 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:46028
I20260202 15:03:27.211949 4378 tablet_replica.cc:333] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: stopping tablet replica
I20260202 15:03:27.212092 4378 raft_consensus.cc:2243] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:27.212380 4378 raft_consensus.cc:2272] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:27.212886 4264 tablet_service.cc:1552] Processing DeleteTablet for tablet 208477fd94c84c2bb5f17499fa1c1890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:27 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:52302
I20260202 15:03:27.213090 4379 tablet_replica.cc:333] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: stopping tablet replica
I20260202 15:03:27.213063 4122 tablet_service.cc:1552] Processing DeleteTablet for tablet 208477fd94c84c2bb5f17499fa1c1890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:27 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60612
I20260202 15:03:27.213171 4378 ts_tablet_manager.cc:1916] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:27.213186 4379 raft_consensus.cc:2243] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:27.213238 4377 tablet_replica.cc:333] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: stopping tablet replica
I20260202 15:03:27.213289 4379 raft_consensus.cc:2272] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:27.213335 4377 raft_consensus.cc:2243] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:27.213438 4377 raft_consensus.cc:2272] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:27.214257 4377 ts_tablet_manager.cc:1916] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:27.215685 4378 ts_tablet_manager.cc:1929] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:27.215783 4378 log.cc:1199] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/208477fd94c84c2bb5f17499fa1c1890
I20260202 15:03:27.216037 4378 ts_tablet_manager.cc:1950] T 208477fd94c84c2bb5f17499fa1c1890 P 7156b28ce5934715b64528f5842d5927: Deleting consensus metadata
I20260202 15:03:27.216673 3825 catalog_manager.cc:4985] TS 7156b28ce5934715b64528f5842d5927 (127.2.164.65:44847): tablet 208477fd94c84c2bb5f17499fa1c1890 (table test-table [id=7cc35a02dfbb4dcb9f33c38e5437e1b9]) successfully deleted
I20260202 15:03:27.216809 4377 ts_tablet_manager.cc:1929] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:27.216887 4377 log.cc:1199] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/208477fd94c84c2bb5f17499fa1c1890
I20260202 15:03:27.217162 4377 ts_tablet_manager.cc:1950] T 208477fd94c84c2bb5f17499fa1c1890 P 92d079bd625948cd82dc9a33cd7b607a: Deleting consensus metadata
I20260202 15:03:27.217499 4379 ts_tablet_manager.cc:1916] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:27.219472 4379 ts_tablet_manager.cc:1929] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:27.219575 4379 log.cc:1199] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/208477fd94c84c2bb5f17499fa1c1890
I20260202 15:03:27.219859 4379 ts_tablet_manager.cc:1950] T 208477fd94c84c2bb5f17499fa1c1890 P f4c12faf72ae4bfb97f67f42db406b52: Deleting consensus metadata
I20260202 15:03:27.220533 3826 catalog_manager.cc:4985] TS f4c12faf72ae4bfb97f67f42db406b52 (127.2.164.67:34511): tablet 208477fd94c84c2bb5f17499fa1c1890 (table test-table [id=7cc35a02dfbb4dcb9f33c38e5437e1b9]) successfully deleted
I20260202 15:03:27.220700 3826 catalog_manager.cc:4985] TS 92d079bd625948cd82dc9a33cd7b607a (127.2.164.66:37205): tablet 208477fd94c84c2bb5f17499fa1c1890 (table test-table [id=7cc35a02dfbb4dcb9f33c38e5437e1b9]) successfully deleted
Feb 02 15:03:27 dist-test-slave-6zzp krb5kdc[3793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:27.233139 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.212135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.2.164.65:44847)
0202 15:03:27.212590 (+ 455us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:27.212597 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:27.220907 (+ 8310us) server_negotiation.cc:408] Connection header received
0202 15:03:27.221026 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:27.221031 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:27.221087 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:27.221173 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:27.221949 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.222690 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.223322 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.223614 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.228972 (+ 5358us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:27.229023 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:27.229042 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:27.229078 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:27.231413 (+ 2335us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:27.231857 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:27.231864 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:27.231866 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:27.231933 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:27.232144 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:27.232147 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:27.232149 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:27.232382 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:27.232587 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:27.232793 (+ 206us) server_negotiation.cc:300] Negotiation successful
0202 15:03:27.232931 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
W20260202 15:03:27.233680 4038 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46060
I20260202 15:03:27.242316 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:27.235437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.2.164.126:34831)
0202 15:03:27.235587 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:27.235592 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:27.235607 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:03:27.235720 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:27.235724 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:27.235778 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:27.235860 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:27.236603 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.237437 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:27.238061 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:27.238248 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:27.239306 (+ 1058us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:27.239326 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:27.239329 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:27.239363 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:27.240778 (+ 1415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:27.241230 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:27.241234 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:27.241236 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:27.241285 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:27.241538 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:27.241541 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:27.241543 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:27.241696 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:27.241839 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:27.242033 (+ 194us) server_negotiation.cc:300] Negotiation successful
0202 15:03:27.242150 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260202 15:03:27.242871 3840 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970
I20260202 15:03:27.244851 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3915
I20260202 15:03:27.251420 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 4057
I20260202 15:03:27.258334 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 4199
I20260202 15:03:27.264353 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 3809
2026-02-02T15:03:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3876 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:27 dist-test-slave-6zzp krb5kdc[4415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:27 dist-test-slave-6zzp krb5kdc[4415](info): set up 2 sockets
Feb 02 15:03:27 dist-test-slave-6zzp krb5kdc[4415](info): commencing operation
krb5kdc: starting...
W20260202 15:03:29.273252 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:29 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:29.413959 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37519
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:29.522421 4431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:29.522677 4431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:29.522778 4431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:29.526027 4431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:29.526108 4431 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260202 15:03:29.526140 4431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:29.526203 4431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:29.526260 4431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:29.526299 4431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:29.530507 4431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:37519
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:29.531599 4431 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:29.532480 4431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:29.538213 4437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:29.538215 4436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:29.538419 4431 server_base.cc:1060] running on GCE node
W20260202 15:03:29.538213 4439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:29.539119 4431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:29.540105 4431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:29.541286 4431 hybrid_clock.cc:648] HybridClock initialized: now 1770044609541264 us; error 46 us; skew 500 ppm
Feb 02 15:03:29 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:29.543920 4431 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:29.545080 4431 webserver.cc:492] Webserver started at http://127.2.164.126:33063/ using document root <none> and password file <none>
I20260202 15:03:29.545663 4431 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:29.545739 4431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:29.545958 4431 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:29.547667 4431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "092aae1ff77b450387e069d2f2e1ddc2"
format_stamp: "Formatted at 2026-02-02 15:03:29 on dist-test-slave-6zzp"
server_key: "e8a9a4530b4359d1d3f3c615d274dca7"
server_key_iv: "6e33a00f006f98a612be31626e1fe118"
server_key_version: "encryptionkey@0"
I20260202 15:03:29.548157 4431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "092aae1ff77b450387e069d2f2e1ddc2"
format_stamp: "Formatted at 2026-02-02 15:03:29 on dist-test-slave-6zzp"
server_key: "e8a9a4530b4359d1d3f3c615d274dca7"
server_key_iv: "6e33a00f006f98a612be31626e1fe118"
server_key_version: "encryptionkey@0"
I20260202 15:03:29.551486 4431 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260202 15:03:29.554066 4446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:29.555104 4431 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:29.555240 4431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "092aae1ff77b450387e069d2f2e1ddc2"
format_stamp: "Formatted at 2026-02-02 15:03:29 on dist-test-slave-6zzp"
server_key: "e8a9a4530b4359d1d3f3c615d274dca7"
server_key_iv: "6e33a00f006f98a612be31626e1fe118"
server_key_version: "encryptionkey@0"
I20260202 15:03:29.555336 4431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:29.579424 4431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:29.582096 4431 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:29.582293 4431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:29.590622 4431 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:37519
I20260202 15:03:29.590633 4508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:37519 every 8 connection(s)
I20260202 15:03:29.591609 4431 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:29.594583 4509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:29.599373 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Bootstrap starting.
I20260202 15:03:29.599581 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 4431
I20260202 15:03:29.599694 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:29.599939 2705 external_mini_cluster.cc:1468] Setting key c2838e79216973fbf9d9ec3ff85ef68d
I20260202 15:03:29.601816 4509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:29.602494 4509 log.cc:826] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:29.604604 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: No bootstrap required, opened a new log
I20260202 15:03:29.607285 4509 raft_consensus.cc:359] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092aae1ff77b450387e069d2f2e1ddc2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37519 } }
I20260202 15:03:29.607492 4509 raft_consensus.cc:385] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:29.607587 4509 raft_consensus.cc:740] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092aae1ff77b450387e069d2f2e1ddc2, State: Initialized, Role: FOLLOWER
Feb 02 15:03:29 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:29.608098 4509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092aae1ff77b450387e069d2f2e1ddc2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37519 } }
I20260202 15:03:29.608289 4509 raft_consensus.cc:399] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:29.608355 4509 raft_consensus.cc:493] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:29.608449 4509 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:29.609663 4509 raft_consensus.cc:515] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092aae1ff77b450387e069d2f2e1ddc2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37519 } }
I20260202 15:03:29.610006 4509 leader_election.cc:304] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092aae1ff77b450387e069d2f2e1ddc2; no voters:
I20260202 15:03:29.610286 4509 leader_election.cc:290] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:29.610397 4514 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:29.610628 4514 raft_consensus.cc:697] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [term 1 LEADER]: Becoming Leader. State: Replica: 092aae1ff77b450387e069d2f2e1ddc2, State: Running, Role: LEADER
I20260202 15:03:29.610941 4514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092aae1ff77b450387e069d2f2e1ddc2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37519 } }
I20260202 15:03:29.611488 4509 sys_catalog.cc:565] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:29.612159 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092aae1ff77b450387e069d2f2e1ddc2. Latest consensus state: current_term: 1 leader_uuid: "092aae1ff77b450387e069d2f2e1ddc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092aae1ff77b450387e069d2f2e1ddc2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37519 } } }
I20260202 15:03:29.612280 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:29.612651 4520 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:29.612902 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092aae1ff77b450387e069d2f2e1ddc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092aae1ff77b450387e069d2f2e1ddc2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37519 } } }
I20260202 15:03:29.613015 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:29.613691 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:29.601248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.2.164.126:37519)
0202 15:03:29.601704 (+ 456us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:29.601712 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:29.601745 (+ 33us) server_negotiation.cc:408] Connection header received
0202 15:03:29.602293 (+ 548us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:29.602309 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:29.602630 (+ 321us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:29.603021 (+ 391us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:29.604055 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.605175 (+ 1120us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:29.605864 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.606134 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:29.608252 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:29.608283 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:29.608294 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:29.608325 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:29.610313 (+ 1988us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:29.610818 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:29.610824 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:29.610829 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:29.610916 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:29.611173 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:29.611176 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:29.611178 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:29.611612 (+ 434us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:29.611849 (+ 237us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:29.612534 (+ 685us) server_negotiation.cc:300] Negotiation successful
0202 15:03:29.612847 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":195,"threads_started":1}
I20260202 15:03:29.615597 4520 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:29.620631 4520 catalog_manager.cc:1357] Generated new cluster ID: b414783434d146c2936551785bb58eee
I20260202 15:03:29.620699 4520 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:29.666968 4520 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:29.667918 4520 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:29.675037 4520 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Generated new TSK 0
I20260202 15:03:29.675674 4520 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:29.735868 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:37519
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:29.858559 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:29.858829 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:29.858932 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:29.862542 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:29.862644 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:29.862732 4537 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260202 15:03:29.862784 4537 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:03:29.862871 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:29.867265 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:29.868467 4537 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:29.869414 4537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:29.876565 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:29.876729 4537 server_base.cc:1060] running on GCE node
W20260202 15:03:29.876595 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:29.876565 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:29.877249 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:29.877882 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:29.879076 4537 hybrid_clock.cc:648] HybridClock initialized: now 1770044609879048 us; error 39 us; skew 500 ppm
Feb 02 15:03:29 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:29.881839 4537 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:29.883270 4537 webserver.cc:492] Webserver started at http://127.2.164.65:43331/ using document root <none> and password file <none>
I20260202 15:03:29.883886 4537 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:29.883967 4537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:29.884184 4537 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:29.885987 4537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "03e2ba39b9944b55b82a546da8748d88"
format_stamp: "Formatted at 2026-02-02 15:03:29 on dist-test-slave-6zzp"
server_key: "4e82f37a2f371f7ee62ef604ee01ef34"
server_key_iv: "3dc24c3a3043fb595d886e8d6c627f1d"
server_key_version: "encryptionkey@0"
I20260202 15:03:29.886488 4537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "03e2ba39b9944b55b82a546da8748d88"
format_stamp: "Formatted at 2026-02-02 15:03:29 on dist-test-slave-6zzp"
server_key: "4e82f37a2f371f7ee62ef604ee01ef34"
server_key_iv: "3dc24c3a3043fb595d886e8d6c627f1d"
server_key_version: "encryptionkey@0"
I20260202 15:03:29.890182 4537 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:03:29.892457 4552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:29.893646 4537 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:29.893787 4537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "03e2ba39b9944b55b82a546da8748d88"
format_stamp: "Formatted at 2026-02-02 15:03:29 on dist-test-slave-6zzp"
server_key: "4e82f37a2f371f7ee62ef604ee01ef34"
server_key_iv: "3dc24c3a3043fb595d886e8d6c627f1d"
server_key_version: "encryptionkey@0"
I20260202 15:03:29.893894 4537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:29.905237 4537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:29.907934 4537 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:29.908139 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:29.909682 4537 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:29.909832 4537 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:29.909952 4537 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:29.910006 4537 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:03:29 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:29.922140 4537 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:44181
I20260202 15:03:29.922225 4667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:44181 every 8 connection(s)
I20260202 15:03:29.923305 4537 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:29.923416 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:29.910391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:33147 (local address 127.2.164.126:37519)
0202 15:03:29.910544 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:29.910549 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:29.911296 (+ 747us) server_negotiation.cc:408] Connection header received
0202 15:03:29.912249 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:29.912253 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:29.912311 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:29.912414 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:29.914451 (+ 2037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.915152 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:29.916137 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.916362 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:29.919265 (+ 2903us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:29.919293 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:29.919296 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:29.919333 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:29.921212 (+ 1879us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:29.921731 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:29.921738 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:29.921743 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:29.921804 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:29.922182 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:29.922188 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:29.922191 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:29.922389 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:29.922492 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:29.923103 (+ 611us) server_negotiation.cc:300] Negotiation successful
0202 15:03:29.923239 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:03:29.923899 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:29.910691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.2.164.65:33147)
0202 15:03:29.911135 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:29.911177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:29.911975 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:29.912599 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:29.912611 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:29.913605 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:29.914295 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:29.914310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.915273 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:29.915280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:29.915981 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:29.915995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.916243 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:29.917216 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:29.917253 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:29.919107 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:29.921331 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:29.921335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:29.921344 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:29.921578 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:29.921925 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:29.921928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:29.921930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:29.922085 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:29.922548 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:29.922554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:29.922852 (+ 298us) client_negotiation.cc:769] Sending connection context
0202 15:03:29.923023 (+ 171us) client_negotiation.cc:240] Negotiation successful
0202 15:03:29.923236 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260202 15:03:29.925710 4668 heartbeater.cc:344] Connected to a master server at 127.2.164.126:37519
I20260202 15:03:29.925871 4668 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:29.926393 4668 heartbeater.cc:507] Master 127.2.164.126:37519 requested a full tablet report, sending...
I20260202 15:03:29.927985 4462 ts_manager.cc:194] Registered new tserver with Master: 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181)
I20260202 15:03:29.929564 4462 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:33147
I20260202 15:03:29.932571 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 4537
I20260202 15:03:29.932736 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:29.932988 2705 external_mini_cluster.cc:1468] Setting key 64a8d950051d3554cc04dc2ec42bc51e
I20260202 15:03:29.934803 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:29.927752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.2.164.65:50025)
0202 15:03:29.928053 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:29.928071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:29.928266 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:29.928530 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:29.928534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:29.928731 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:29.929050 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:29.929060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.930027 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:29.930031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:29.930442 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:29.930450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.930581 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:29.931193 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:29.931206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:29.931484 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:29.933406 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:29.933410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:29.933413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:29.933598 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:29.933856 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:29.933858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:29.933860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:29.933923 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:29.934263 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:29.934267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:29.934354 (+ 87us) client_negotiation.cc:769] Sending connection context
0202 15:03:29.934465 (+ 111us) client_negotiation.cc:240] Negotiation successful
0202 15:03:29.934618 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260202 15:03:29.934823 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:29.927883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50025 (local address 127.2.164.126:37519)
0202 15:03:29.927996 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:29.927999 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:29.928271 (+ 272us) server_negotiation.cc:408] Connection header received
0202 15:03:29.928377 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:29.928379 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:29.928422 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:29.928492 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:29.929248 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.929911 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:29.930562 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:29.930775 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:29.931622 (+ 847us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:29.931642 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:29.931644 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:29.931665 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:29.933303 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:29.933697 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:29.933702 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:29.933704 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:29.933750 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:29.934015 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:29.934017 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:29.934019 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:29.934165 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:29.934251 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:29.934538 (+ 287us) server_negotiation.cc:300] Negotiation successful
0202 15:03:29.934667 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:29.982247 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:37519
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:30.089371 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:30.089603 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:30.089699 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:30.092928 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:30.093044 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:30.093163 4679 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260202 15:03:30.093206 4679 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:03:30.093283 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:30.097474 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:30.098585 4679 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:30.099472 4679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:30.106036 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:30.106087 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:30.106118 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:30.106200 4679 server_base.cc:1060] running on GCE node
I20260202 15:03:30.106721 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:30.107275 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:30.108438 4679 hybrid_clock.cc:648] HybridClock initialized: now 1770044610108428 us; error 24 us; skew 500 ppm
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044610, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:30.111099 4679 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:30.112112 4679 webserver.cc:492] Webserver started at http://127.2.164.66:38561/ using document root <none> and password file <none>
I20260202 15:03:30.112841 4679 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:30.112895 4679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:30.113111 4679 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:30.114837 4679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "ff23af135c614f90bd96dbce69363847"
format_stamp: "Formatted at 2026-02-02 15:03:30 on dist-test-slave-6zzp"
server_key: "28eb8a997e10232e187567d44a877032"
server_key_iv: "5142134d10f94cfabc16e6702085a722"
server_key_version: "encryptionkey@0"
I20260202 15:03:30.115319 4679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "ff23af135c614f90bd96dbce69363847"
format_stamp: "Formatted at 2026-02-02 15:03:30 on dist-test-slave-6zzp"
server_key: "28eb8a997e10232e187567d44a877032"
server_key_iv: "5142134d10f94cfabc16e6702085a722"
server_key_version: "encryptionkey@0"
I20260202 15:03:30.118794 4679 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:30.120839 4694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:30.121872 4679 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:30.121989 4679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "ff23af135c614f90bd96dbce69363847"
format_stamp: "Formatted at 2026-02-02 15:03:30 on dist-test-slave-6zzp"
server_key: "28eb8a997e10232e187567d44a877032"
server_key_iv: "5142134d10f94cfabc16e6702085a722"
server_key_version: "encryptionkey@0"
I20260202 15:03:30.122077 4679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:30.145109 4679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:30.147794 4679 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:30.147984 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:30.149499 4679 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:30.149580 4679 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:30.149771 4679 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:30.149847 4679 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044610, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:30.161103 4679 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:41013
I20260202 15:03:30.161147 4809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:41013 every 8 connection(s)
I20260202 15:03:30.162074 4679 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:30.162217 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.150097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:59765 (local address 127.2.164.126:37519)
0202 15:03:30.150229 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.150234 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.150978 (+ 744us) server_negotiation.cc:408] Connection header received
0202 15:03:30.151823 (+ 845us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.151827 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.151889 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.151997 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:30.153502 (+ 1505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.154338 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.155001 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.155218 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.157727 (+ 2509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:30.157756 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:30.157760 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:30.157818 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:30.159977 (+ 2159us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.160516 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.160520 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.160522 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.160586 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.160962 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.160965 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.160966 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.161234 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:30.161371 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.161897 (+ 526us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.162041 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260202 15:03:30.162950 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.150387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.2.164.66:59765)
0202 15:03:30.150812 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.150853 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.151612 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.152169 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.152177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.152644 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:30.153340 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.153353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.154474 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.154481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.154905 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.154913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.155153 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.155823 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:30.155865 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:30.157572 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:30.160111 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.160117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.160131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.160401 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.160696 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.160699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.160701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.160840 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.161369 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:30.161376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:30.161645 (+ 269us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.161857 (+ 212us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.162136 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":129,"threads_started":1}
I20260202 15:03:30.164075 4810 heartbeater.cc:344] Connected to a master server at 127.2.164.126:37519
I20260202 15:03:30.164225 4810 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:30.164813 4810 heartbeater.cc:507] Master 127.2.164.126:37519 requested a full tablet report, sending...
I20260202 15:03:30.165901 4463 ts_manager.cc:194] Registered new tserver with Master: ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013)
I20260202 15:03:30.166639 4463 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:59765
I20260202 15:03:30.167840 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 4679
I20260202 15:03:30.167939 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:30.168182 2705 external_mini_cluster.cc:1468] Setting key 02c1a0b3543a0904325f4dfe60ad5a18
I20260202 15:03:30.172969 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.166307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.2.164.66:42255)
0202 15:03:30.167016 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.167030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.167136 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.167380 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.167382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.167527 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:30.167732 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.167737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.168548 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.168554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.169053 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.169063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.169186 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.169667 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:30.169679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:30.170066 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:30.171657 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.171660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.171662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.171823 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.172088 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.172092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.172093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.172135 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.172579 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:30.172583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:30.172657 (+ 74us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.172751 (+ 94us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.172848 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":96,"threads_started":1}
I20260202 15:03:30.173195 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.166366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42255 (local address 127.2.164.126:37519)
0202 15:03:30.166492 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.166496 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.167114 (+ 618us) server_negotiation.cc:408] Connection header received
0202 15:03:30.167228 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.167231 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.167276 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.167356 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:30.167859 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.168444 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.169170 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.169378 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.170190 (+ 812us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:30.170203 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:30.170205 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:30.170226 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:30.171552 (+ 1326us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.171929 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.171933 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.171936 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.171985 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.172227 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.172230 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.172232 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.172443 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:30.172556 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.172833 (+ 277us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.172955 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:30.220254 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:37519
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:30.329540 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:30.329777 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:30.329871 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:30.333144 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:30.333225 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:30.333309 4821 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260202 15:03:30.333355 4821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:03:30.333436 4821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:30.337630 4821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:37519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:30.338738 4821 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:30.339587 4821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:30.346450 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:30.346526 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:30.346527 4827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:30.346757 4821 server_base.cc:1060] running on GCE node
I20260202 15:03:30.347131 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:30.347667 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:30.348847 4821 hybrid_clock.cc:648] HybridClock initialized: now 1770044610348827 us; error 43 us; skew 500 ppm
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044610, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:30.351701 4821 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:30.352744 4821 webserver.cc:492] Webserver started at http://127.2.164.67:46453/ using document root <none> and password file <none>
I20260202 15:03:30.353345 4821 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:30.353416 4821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:30.353627 4821 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:30.355357 4821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "b5c7b4cef7974153b7c0290b29e6733f"
format_stamp: "Formatted at 2026-02-02 15:03:30 on dist-test-slave-6zzp"
server_key: "7b6ac8ca0f0e12c4e91021668f7af1ad"
server_key_iv: "dac018092d5ed9a7e51d1889198aad78"
server_key_version: "encryptionkey@0"
I20260202 15:03:30.355850 4821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c7b4cef7974153b7c0290b29e6733f"
format_stamp: "Formatted at 2026-02-02 15:03:30 on dist-test-slave-6zzp"
server_key: "7b6ac8ca0f0e12c4e91021668f7af1ad"
server_key_iv: "dac018092d5ed9a7e51d1889198aad78"
server_key_version: "encryptionkey@0"
I20260202 15:03:30.359230 4821 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260202 15:03:30.361635 4836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:30.362815 4821 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:30.362965 4821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "b5c7b4cef7974153b7c0290b29e6733f"
format_stamp: "Formatted at 2026-02-02 15:03:30 on dist-test-slave-6zzp"
server_key: "7b6ac8ca0f0e12c4e91021668f7af1ad"
server_key_iv: "dac018092d5ed9a7e51d1889198aad78"
server_key_version: "encryptionkey@0"
I20260202 15:03:30.363075 4821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:30.385892 4821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:30.388455 4821 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:30.388659 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:30.390203 4821 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:30.390268 4821 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:30.390321 4821 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:30.390349 4821 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044610, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:30.401772 4821 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43581
I20260202 15:03:30.401918 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.390714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:52343 (local address 127.2.164.126:37519)
0202 15:03:30.390857 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.390860 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.391626 (+ 766us) server_negotiation.cc:408] Connection header received
0202 15:03:30.392472 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.392476 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.392535 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.392623 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:30.394108 (+ 1485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.394609 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.395301 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.395478 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.398274 (+ 2796us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:30.398301 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:30.398305 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:30.398360 (+ 55us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:30.399905 (+ 1545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.400384 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.400387 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.400389 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.400433 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.400730 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.400734 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.400736 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.400933 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:30.401063 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.401604 (+ 541us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.401739 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260202 15:03:30.401899 4951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:43581 every 8 connection(s)
I20260202 15:03:30.402562 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.391034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.2.164.67:52343)
0202 15:03:30.391484 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.391518 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.392279 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.392786 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.392795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.393298 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:30.393961 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.393974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.394727 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.394731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.395183 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.395190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.395413 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.396005 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:30.396028 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:30.398092 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:30.400022 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.400027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.400038 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.400268 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.400531 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.400534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.400535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.400617 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.401075 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:30.401080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:30.401298 (+ 218us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.401539 (+ 241us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.401854 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":132,"threads_started":1}
I20260202 15:03:30.402791 4821 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:30.404518 4952 heartbeater.cc:344] Connected to a master server at 127.2.164.126:37519
I20260202 15:03:30.404701 4952 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:30.405534 4952 heartbeater.cc:507] Master 127.2.164.126:37519 requested a full tablet report, sending...
I20260202 15:03:30.406205 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 4821
I20260202 15:03:30.406325 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:30.406728 2705 external_mini_cluster.cc:1468] Setting key 5140e2e0252438eec33a0b4ca550db87
I20260202 15:03:30.407461 4463 ts_manager.cc:194] Registered new tserver with Master: b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:30.408210 4463 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:52343
I20260202 15:03:30.408751 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260202 15:03:30.413473 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.406099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.2.164.67:45249)
0202 15:03:30.406399 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.406416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.406530 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.407499 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.407501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.407785 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:30.408069 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.408076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.409033 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.409039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.409423 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.409430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.409559 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.410147 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:30.410159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:30.410423 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:30.412063 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.412066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.412068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.412237 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.412474 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.412479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.412481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.412530 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.412872 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:30.412876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:30.412982 (+ 106us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.413180 (+ 198us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.413301 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260202 15:03:30.413523 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.407040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45249 (local address 127.2.164.126:37519)
0202 15:03:30.407182 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.407186 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.407198 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:03:30.407230 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.407233 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.407281 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.407387 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:30.408178 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.408907 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.409546 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.409751 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.410545 (+ 794us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:30.410561 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:30.410565 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:30.410588 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:30.411934 (+ 1346us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.412334 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.412338 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.412340 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.412387 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.412630 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.412633 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.412635 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.412768 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:30.412883 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.413266 (+ 383us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.413402 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:30.432935 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.425516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.2.164.126:37519)
0202 15:03:30.425685 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.425689 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.425701 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:03:30.425736 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.425738 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.425794 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.425903 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:30.426802 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.427284 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.427896 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.428050 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.430087 (+ 2037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:30.430109 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:30.430112 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:30.430134 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:30.431452 (+ 1318us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.431867 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.431870 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.431871 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.431912 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.432138 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.432142 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.432144 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.432291 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:30.432425 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.432644 (+ 219us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.432756 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260202 15:03:30.435535 4462 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:30.437892 4462 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:30.449879 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.445626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41013 (local address 127.0.0.1:47958)
0202 15:03:30.446459 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.446500 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.446662 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.447254 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.447260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.447283 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:30.447529 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.447536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.448615 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.448620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.449407 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.449415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.449521 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.449585 (+ 64us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.449679 (+ 94us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.449776 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":67,"threads_started":1}
I20260202 15:03:30.450506 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.445766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43581 (local address 127.0.0.1:54624)
0202 15:03:30.446549 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.446565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.446664 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.447340 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.447343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.447360 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:30.447627 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.447635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.449154 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.449159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.450194 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.450203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.450273 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.450290 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.450331 (+ 41us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.450386 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":87,"threads_started":1}
I20260202 15:03:30.450824 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.446276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44181 (local address 127.0.0.1:60102)
0202 15:03:30.446818 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.446831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.447019 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.447625 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.447628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.447650 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:30.447888 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.447895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.449375 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.449378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.450286 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.450293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.450379 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.450390 (+ 11us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.450425 (+ 35us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.450468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":75,"threads_started":1}
I20260202 15:03:30.450872 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.445765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.2.164.66:41013)
0202 15:03:30.446129 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.446135 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.446796 (+ 661us) server_negotiation.cc:408] Connection header received
0202 15:03:30.446888 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.446895 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.447085 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.447244 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:30.447678 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.448501 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.450085 (+ 1584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.450473 (+ 388us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.450599 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.450669 (+ 70us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.450745 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260202 15:03:30.451659 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.446370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.2.164.67:43581)
0202 15:03:30.446924 (+ 554us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.446932 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.446955 (+ 23us) server_negotiation.cc:408] Connection header received
0202 15:03:30.447031 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.447037 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.447194 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.447761 (+ 567us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:30.447885 (+ 124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.449023 (+ 1138us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.450824 (+ 1801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.451295 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.451386 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.451440 (+ 54us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.451511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":93,"threads_started":1}
I20260202 15:03:30.452173 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.446411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.2.164.65:44181)
0202 15:03:30.447249 (+ 838us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.447257 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.447279 (+ 22us) server_negotiation.cc:408] Connection header received
0202 15:03:30.447356 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.447361 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.447502 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.447624 (+ 122us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:30.448020 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.449136 (+ 1116us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.450626 (+ 1490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.451831 (+ 1205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.451931 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.451984 (+ 53us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.452058 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":83,"threads_started":1}
I20260202 15:03:30.453751 4744 tablet_service.cc:1505] Processing CreateTablet for tablet 4501fc26d912471080c3bda747517327 (DEFAULT_TABLE table=test-table [id=bf4b2ecf019f4e14bc0edfc2d702157f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:30.454102 4886 tablet_service.cc:1505] Processing CreateTablet for tablet 4501fc26d912471080c3bda747517327 (DEFAULT_TABLE table=test-table [id=bf4b2ecf019f4e14bc0edfc2d702157f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:30.454383 4602 tablet_service.cc:1505] Processing CreateTablet for tablet 4501fc26d912471080c3bda747517327 (DEFAULT_TABLE table=test-table [id=bf4b2ecf019f4e14bc0edfc2d702157f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:30.454676 4744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4501fc26d912471080c3bda747517327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.454962 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4501fc26d912471080c3bda747517327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.455142 4602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4501fc26d912471080c3bda747517327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.460556 4975 tablet_bootstrap.cc:492] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Bootstrap starting.
I20260202 15:03:30.461946 4977 tablet_bootstrap.cc:492] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Bootstrap starting.
I20260202 15:03:30.461969 4976 tablet_bootstrap.cc:492] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Bootstrap starting.
I20260202 15:03:30.463227 4975 tablet_bootstrap.cc:654] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.464083 4977 tablet_bootstrap.cc:654] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.464083 4976 tablet_bootstrap.cc:654] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.464318 4975 log.cc:826] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:30.464975 4976 log.cc:826] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:30.465391 4977 log.cc:826] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:30.466840 4976 tablet_bootstrap.cc:492] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: No bootstrap required, opened a new log
I20260202 15:03:30.466912 4975 tablet_bootstrap.cc:492] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: No bootstrap required, opened a new log
I20260202 15:03:30.467104 4975 ts_tablet_manager.cc:1403] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260202 15:03:30.467108 4976 ts_tablet_manager.cc:1403] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:30.467358 4977 tablet_bootstrap.cc:492] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: No bootstrap required, opened a new log
I20260202 15:03:30.467547 4977 ts_tablet_manager.cc:1403] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:03:30.469956 4975 raft_consensus.cc:359] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.470165 4975 raft_consensus.cc:385] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.470218 4975 raft_consensus.cc:740] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff23af135c614f90bd96dbce69363847, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.470316 4977 raft_consensus.cc:359] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.470316 4976 raft_consensus.cc:359] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.470505 4977 raft_consensus.cc:385] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.470505 4976 raft_consensus.cc:385] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.470561 4977 raft_consensus.cc:740] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c7b4cef7974153b7c0290b29e6733f, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.470561 4976 raft_consensus.cc:740] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.470716 4975 consensus_queue.cc:260] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.470964 4977 consensus_queue.cc:260] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.470964 4976 consensus_queue.cc:260] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.471478 4810 heartbeater.cc:499] Master 127.2.164.126:37519 was elected leader, sending a full tablet report...
I20260202 15:03:30.471673 4975 ts_tablet_manager.cc:1434] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:03:30.471707 4977 ts_tablet_manager.cc:1434] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260202 15:03:30.472020 4952 heartbeater.cc:499] Master 127.2.164.126:37519 was elected leader, sending a full tablet report...
I20260202 15:03:30.472769 4976 ts_tablet_manager.cc:1434] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:03:30.473119 4668 heartbeater.cc:499] Master 127.2.164.126:37519 was elected leader, sending a full tablet report...
I20260202 15:03:30.554934 4982 raft_consensus.cc:493] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:30.555176 4982 raft_consensus.cc:515] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.556408 4982 leader_election.cc:290] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:30.559788 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.556715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41013 (local address 127.2.164.65:44651)
0202 15:03:30.557158 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.557171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.557312 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.557599 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.557602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.557622 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:30.557883 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.557890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.558729 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.558732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.559306 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.559313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.559450 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.559471 (+ 21us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.559535 (+ 64us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.559598 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":690,"threads_started":1}
I20260202 15:03:30.560259 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.556816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:44651 (local address 127.2.164.66:41013)
0202 15:03:30.556959 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.556964 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.557255 (+ 291us) server_negotiation.cc:408] Connection header received
0202 15:03:30.557419 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.557425 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.557478 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.557585 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:30.558004 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.558596 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.559445 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.559963 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.560005 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.560084 (+ 79us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.560146 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260202 15:03:30.560387 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.556715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43581 (local address 127.2.164.65:48837)
0202 15:03:30.557606 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.557619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.557748 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.558026 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.558029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.558045 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:30.558338 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.558347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.559125 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.559131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.559929 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.559938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.560104 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.560126 (+ 22us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.560181 (+ 55us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.560228 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":52,"thread_start_us":172,"threads_started":1}
I20260202 15:03:30.560858 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.556817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:48837 (local address 127.2.164.67:43581)
0202 15:03:30.556959 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.556964 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.557705 (+ 741us) server_negotiation.cc:408] Connection header received
0202 15:03:30.557863 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.557869 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.557919 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.557999 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:30.558518 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.559007 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.560109 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.560608 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.560645 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.560694 (+ 49us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.560749 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260202 15:03:30.560878 4764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fc26d912471080c3bda747517327" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff23af135c614f90bd96dbce69363847" is_pre_election: true
I20260202 15:03:30.561209 4764 raft_consensus.cc:2468] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 0.
I20260202 15:03:30.561473 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fc26d912471080c3bda747517327" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f" is_pre_election: true
I20260202 15:03:30.561725 4903 raft_consensus.cc:2468] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 0.
I20260202 15:03:30.561723 4556 leader_election.cc:304] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:30.562032 4982 raft_consensus.cc:2804] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:30.562117 4982 raft_consensus.cc:493] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:30.562168 4982 raft_consensus.cc:3060] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:30.563115 4982 raft_consensus.cc:515] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.563467 4982 leader_election.cc:290] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 election: Requested vote from peers ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:30.563815 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fc26d912471080c3bda747517327" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f"
I20260202 15:03:30.563866 4764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fc26d912471080c3bda747517327" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff23af135c614f90bd96dbce69363847"
I20260202 15:03:30.563915 4903 raft_consensus.cc:3060] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:30.564021 4764 raft_consensus.cc:3060] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:30.565092 4903 raft_consensus.cc:2468] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 1.
I20260202 15:03:30.565196 4764 raft_consensus.cc:2468] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 1.
I20260202 15:03:30.565449 4555 leader_election.cc:304] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, b5c7b4cef7974153b7c0290b29e6733f; no voters:
I20260202 15:03:30.565650 4982 raft_consensus.cc:2804] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:30.565903 4982 raft_consensus.cc:697] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 1 LEADER]: Becoming Leader. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Running, Role: LEADER
I20260202 15:03:30.566239 4982 consensus_queue.cc:237] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } }
I20260202 15:03:30.569473 4460 catalog_manager.cc:5654] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } health_report { overall_health: HEALTHY } } }
Feb 02 15:03:30 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:30.622681 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.614882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.2.164.126:37519)
0202 15:03:30.615042 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.615046 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.615263 (+ 217us) server_negotiation.cc:408] Connection header received
0202 15:03:30.615373 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.615376 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.615419 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.615489 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:30.616408 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.616914 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.617608 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.617742 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.619870 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:30.619886 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:30.619888 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:30.619909 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:30.621226 (+ 1317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.621673 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.621675 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.621677 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.621716 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:30.621973 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:30.621975 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:30.621977 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:30.622109 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:30.622184 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.622436 (+ 252us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.622552 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":53}
I20260202 15:03:30.622682 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.614860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37519 (local address 127.0.0.1:32868)
0202 15:03:30.615172 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.615185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.615289 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.615572 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.615576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.616026 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:30.616247 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.616254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.617095 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.617098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.617475 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.617482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.617589 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.618216 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:30.618231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:30.619751 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:30.621337 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.621345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.621350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.621584 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.621825 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:30.621828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:30.621829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:30.621884 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:30.622200 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:30.622205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:30.622294 (+ 89us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.622383 (+ 89us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.622493 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":137,"threads_started":1}
I20260202 15:03:30.626438 4460 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:32868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260202 15:03:30.627219 4460 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:30.632478 4744 tablet_service.cc:1505] Processing CreateTablet for tablet d6933693288b4add9285b4e3623a5161 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c524ccdc23240b780b8a3a32ddbfc61]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:03:30.632472 4602 tablet_service.cc:1505] Processing CreateTablet for tablet d6933693288b4add9285b4e3623a5161 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c524ccdc23240b780b8a3a32ddbfc61]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:03:30.633149 4744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6933693288b4add9285b4e3623a5161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.633147 4602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6933693288b4add9285b4e3623a5161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.632877 4886 tablet_service.cc:1505] Processing CreateTablet for tablet d6933693288b4add9285b4e3623a5161 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c524ccdc23240b780b8a3a32ddbfc61]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:03:30.633579 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6933693288b4add9285b4e3623a5161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.635972 4975 tablet_bootstrap.cc:492] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: Bootstrap starting.
I20260202 15:03:30.636225 4977 tablet_bootstrap.cc:492] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: Bootstrap starting.
I20260202 15:03:30.637113 4977 tablet_bootstrap.cc:654] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.637135 4975 tablet_bootstrap.cc:654] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.637176 4976 tablet_bootstrap.cc:492] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: Bootstrap starting.
I20260202 15:03:30.638266 4976 tablet_bootstrap.cc:654] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.638386 4977 tablet_bootstrap.cc:492] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: No bootstrap required, opened a new log
I20260202 15:03:30.638453 4975 tablet_bootstrap.cc:492] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: No bootstrap required, opened a new log
I20260202 15:03:30.638463 4977 ts_tablet_manager.cc:1403] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260202 15:03:30.638511 4975 ts_tablet_manager.cc:1403] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260202 15:03:30.638919 4975 raft_consensus.cc:359] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.638964 4977 raft_consensus.cc:359] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.639057 4975 raft_consensus.cc:385] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.639086 4977 raft_consensus.cc:385] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.639097 4975 raft_consensus.cc:740] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff23af135c614f90bd96dbce69363847, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.639130 4977 raft_consensus.cc:740] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c7b4cef7974153b7c0290b29e6733f, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.639232 4975 consensus_queue.cc:260] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.639272 4977 consensus_queue.cc:260] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.639513 4975 ts_tablet_manager.cc:1434] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:03:30.639485 4981 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.639540 4977 ts_tablet_manager.cc:1434] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:03:30.639580 4976 tablet_bootstrap.cc:492] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: No bootstrap required, opened a new log
I20260202 15:03:30.639585 4981 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:30.639648 4976 ts_tablet_manager.cc:1403] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:03:30.639981 4983 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.640095 4983 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:30.640103 4976 raft_consensus.cc:359] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.640179 4976 raft_consensus.cc:385] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.640236 4976 raft_consensus.cc:740] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.640357 4976 consensus_queue.cc:260] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.640631 4976 ts_tablet_manager.cc:1434] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:30.640592 4982 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.640709 4982 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: This TxnStatusTablet replica's current role is: FOLLOWER
W20260202 15:03:30.653692 4953 tablet.cc:2378] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260202 15:03:30.663527 4811 tablet.cc:2378] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260202 15:03:30.675026 4669 tablet.cc:2378] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:03:30.825408 4982 raft_consensus.cc:493] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:30.825604 4982 raft_consensus.cc:515] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.826124 4982 leader_election.cc:290] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:30.826556 4764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6933693288b4add9285b4e3623a5161" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff23af135c614f90bd96dbce69363847" is_pre_election: true
I20260202 15:03:30.826635 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6933693288b4add9285b4e3623a5161" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f" is_pre_election: true
I20260202 15:03:30.826714 4764 raft_consensus.cc:2468] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 0.
I20260202 15:03:30.826820 4903 raft_consensus.cc:2468] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 0.
I20260202 15:03:30.827101 4556 leader_election.cc:304] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:30.827308 4982 raft_consensus.cc:2804] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:30.827373 4982 raft_consensus.cc:493] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:30.827401 4982 raft_consensus.cc:3060] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:30.828262 4982 raft_consensus.cc:515] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.828755 4982 leader_election.cc:290] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 election: Requested vote from peers ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:30.829097 4764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6933693288b4add9285b4e3623a5161" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff23af135c614f90bd96dbce69363847"
I20260202 15:03:30.829191 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6933693288b4add9285b4e3623a5161" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f"
I20260202 15:03:30.829253 4764 raft_consensus.cc:3060] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:30.829320 4903 raft_consensus.cc:3060] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:30.830188 4903 raft_consensus.cc:2468] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 1.
I20260202 15:03:30.830191 4764 raft_consensus.cc:2468] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 1.
I20260202 15:03:30.830487 4555 leader_election.cc:304] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, b5c7b4cef7974153b7c0290b29e6733f; no voters:
I20260202 15:03:30.830678 4982 raft_consensus.cc:2804] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:30.830789 4982 raft_consensus.cc:697] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [term 1 LEADER]: Becoming Leader. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Running, Role: LEADER
I20260202 15:03:30.830945 4982 consensus_queue.cc:237] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.831660 4986 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: TxnStatusTablet state changed. Reason: New leader 03e2ba39b9944b55b82a546da8748d88. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.831749 4986 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:03:30.832322 4998 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260202 15:03:30.832572 4998 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260202 15:03:30.832901 4460 catalog_manager.cc:5654] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } health_report { overall_health: UNKNOWN } } }
I20260202 15:03:30.878262 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.874743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.2.164.65:44181)
0202 15:03:30.874897 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.874902 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.874917 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:03:30.875058 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.875062 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.875120 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.875444 (+ 324us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:30.875804 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.876305 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.877155 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.877366 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.877418 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:30.877856 (+ 438us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:30.877984 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.878072 (+ 88us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.878124 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260202 15:03:30.878262 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.874608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44181 (local address 127.0.0.1:60106)
0202 15:03:30.874765 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.874784 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.874903 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.875444 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.875448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.875462 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:03:30.875684 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.875691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.876453 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.876462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.876981 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.876989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.877159 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.877207 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:03:30.877960 (+ 753us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:03:30.877977 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.878052 (+ 75us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.878131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260202 15:03:30.883087 4764 raft_consensus.cc:1275] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Refusing update from remote peer 03e2ba39b9944b55b82a546da8748d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:30.883097 4903 raft_consensus.cc:1275] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Refusing update from remote peer 03e2ba39b9944b55b82a546da8748d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:30.883731 4982 consensus_queue.cc:1048] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:30.883898 4986 consensus_queue.cc:1048] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:30.887933 4983 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: TxnStatusTablet state changed. Reason: New leader 03e2ba39b9944b55b82a546da8748d88. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.888047 4983 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:30.888083 4981 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: TxnStatusTablet state changed. Reason: New leader 03e2ba39b9944b55b82a546da8748d88. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.888183 4981 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:30.891211 4986 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.891321 4986 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:03:30.891662 5002 mvcc.cc:204] Tried to move back new op lower bound from 7250102726170116096 to 7250102725964763136. Current Snapshot: MvccSnapshot[applied={T|T < 7250102726170116096}]
I20260202 15:03:30.891866 4981 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.891950 4981 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P ff23af135c614f90bd96dbce69363847: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:30.894845 4982 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.894960 4982 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P 03e2ba39b9944b55b82a546da8748d88: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:03:30.895814 4983 tablet_replica.cc:442] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } } }
I20260202 15:03:30.895922 4983 tablet_replica.cc:445] T d6933693288b4add9285b4e3623a5161 P b5c7b4cef7974153b7c0290b29e6733f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:03:30.896953 5003 mvcc.cc:204] Tried to move back new op lower bound from 7250102726170116096 to 7250102725964763136. Current Snapshot: MvccSnapshot[applied={T|T < 7250102726170116096}]
I20260202 15:03:30.901412 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.898773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.2.164.65:44181)
0202 15:03:30.898894 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.898899 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.898913 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:03:30.898969 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.898972 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.899022 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.899101 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:30.899452 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.900029 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.900681 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.900873 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.900914 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:30.900980 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:30.901116 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.901229 (+ 113us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.901277 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260202 15:03:30.911787 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.909031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:54477 (local address 127.2.164.65:44181)
0202 15:03:30.909137 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:30.909140 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:30.909328 (+ 188us) server_negotiation.cc:408] Connection header received
0202 15:03:30.909481 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:30.909485 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:30.909547 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:30.909629 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:30.909976 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.910426 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.911119 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.911287 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.911336 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:30.911396 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:30.911472 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:30.911586 (+ 114us) server_negotiation.cc:300] Negotiation successful
0202 15:03:30.911623 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260202 15:03:30.911803 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:30.908921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44181 (local address 127.2.164.65:54477)
0202 15:03:30.909246 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:30.909257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:30.909383 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:30.909635 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:30.909638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:30.909645 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:03:30.909876 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.909882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.910535 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:30.910539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:30.910958 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:30.910965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:30.911135 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:30.911179 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:03:30.911488 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:03:30.911493 (+ 5us) client_negotiation.cc:769] Sending connection context
0202 15:03:30.911577 (+ 84us) client_negotiation.cc:240] Negotiation successful
0202 15:03:30.911638 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260202 15:03:30.919849 4764 raft_consensus.cc:1275] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Refusing update from remote peer 03e2ba39b9944b55b82a546da8748d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:30.920322 4986 consensus_queue.cc:1048] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:30.920398 4903 raft_consensus.cc:1275] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Refusing update from remote peer 03e2ba39b9944b55b82a546da8748d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:30.920805 4986 consensus_queue.cc:1048] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:30.923905 5001 mvcc.cc:204] Tried to move back new op lower bound from 7250102726323580928 to 7250102724882169856. Current Snapshot: MvccSnapshot[applied={T|T < 7250102726323580928}]
I20260202 15:03:30.971385 4460 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:30.971573 4460 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:30.974146 4460 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Sending DeleteTablet for 3 replicas of tablet 4501fc26d912471080c3bda747517327
I20260202 15:03:30.974876 4744 tablet_service.cc:1552] Processing DeleteTablet for tablet 4501fc26d912471080c3bda747517327 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:30 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47958
I20260202 15:03:30.974972 4886 tablet_service.cc:1552] Processing DeleteTablet for tablet 4501fc26d912471080c3bda747517327 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:30 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:54624
I20260202 15:03:30.975595 4602 tablet_service.cc:1552] Processing DeleteTablet for tablet 4501fc26d912471080c3bda747517327 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:30 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60102
I20260202 15:03:30.975833 4460 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260202 15:03:30.976029 5018 tablet_replica.cc:333] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: stopping tablet replica
I20260202 15:03:30.976203 5020 tablet_replica.cc:333] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: stopping tablet replica
W20260202 15:03:30.976269 4460 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:30.976312 5018 raft_consensus.cc:2243] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:30.976559 5018 raft_consensus.cc:2272] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:30.976886 5019 tablet_replica.cc:333] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: stopping tablet replica
I20260202 15:03:30.977066 5020 raft_consensus.cc:2243] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:30.977154 5019 raft_consensus.cc:2243] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:30.977356 5019 raft_consensus.cc:2272] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:30.977576 5020 raft_consensus.cc:2272] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:30.978606 5020 ts_tablet_manager.cc:1916] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:30.978606 5019 ts_tablet_manager.cc:1916] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:30.981388 5020 ts_tablet_manager.cc:1929] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:30.981388 5019 ts_tablet_manager.cc:1929] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:30.981479 5019 log.cc:1199] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/4501fc26d912471080c3bda747517327
I20260202 15:03:30.981611 5020 log.cc:1199] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/4501fc26d912471080c3bda747517327
I20260202 15:03:30.981827 5019 ts_tablet_manager.cc:1950] T 4501fc26d912471080c3bda747517327 P b5c7b4cef7974153b7c0290b29e6733f: Deleting consensus metadata
I20260202 15:03:30.981961 4602 tablet_service.cc:1505] Processing CreateTablet for tablet 3bb4c2ec46754fbb8c2e65face473c4f (DEFAULT_TABLE table=test-table [id=9d95dc62c92c4ba48d60874ae5b219f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:30.982041 5020 ts_tablet_manager.cc:1950] T 4501fc26d912471080c3bda747517327 P 03e2ba39b9944b55b82a546da8748d88: Deleting consensus metadata
I20260202 15:03:30.982232 4602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb4c2ec46754fbb8c2e65face473c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.983105 4450 catalog_manager.cc:4985] TS 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181): tablet 4501fc26d912471080c3bda747517327 (table test-table [id=bf4b2ecf019f4e14bc0edfc2d702157f]) successfully deleted
I20260202 15:03:30.984397 4976 tablet_bootstrap.cc:492] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Bootstrap starting.
I20260202 15:03:30.984555 4744 tablet_service.cc:1505] Processing CreateTablet for tablet 3bb4c2ec46754fbb8c2e65face473c4f (DEFAULT_TABLE table=test-table [id=9d95dc62c92c4ba48d60874ae5b219f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:30.984859 4744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb4c2ec46754fbb8c2e65face473c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.985260 4449 catalog_manager.cc:4985] TS b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581): tablet 4501fc26d912471080c3bda747517327 (table test-table [id=bf4b2ecf019f4e14bc0edfc2d702157f]) successfully deleted
I20260202 15:03:30.985648 4976 tablet_bootstrap.cc:654] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.986480 4886 tablet_service.cc:1505] Processing CreateTablet for tablet 3bb4c2ec46754fbb8c2e65face473c4f (DEFAULT_TABLE table=test-table [id=9d95dc62c92c4ba48d60874ae5b219f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:30.986972 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb4c2ec46754fbb8c2e65face473c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:30.987293 4975 tablet_bootstrap.cc:492] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Bootstrap starting.
I20260202 15:03:30.987550 5018 ts_tablet_manager.cc:1916] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:30.988272 4975 tablet_bootstrap.cc:654] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.989135 4977 tablet_bootstrap.cc:492] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Bootstrap starting.
I20260202 15:03:30.989365 4976 tablet_bootstrap.cc:492] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: No bootstrap required, opened a new log
I20260202 15:03:30.989464 4976 ts_tablet_manager.cc:1403] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260202 15:03:30.990005 4976 raft_consensus.cc:359] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.990108 4977 tablet_bootstrap.cc:654] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:30.990123 4976 raft_consensus.cc:385] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.990155 4976 raft_consensus.cc:740] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.990309 4976 consensus_queue.cc:260] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.990614 4976 ts_tablet_manager.cc:1434] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:03:30.991549 4975 tablet_bootstrap.cc:492] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: No bootstrap required, opened a new log
I20260202 15:03:30.991653 4975 ts_tablet_manager.cc:1403] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260202 15:03:30.992154 4975 raft_consensus.cc:359] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.992277 4975 raft_consensus.cc:385] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.992313 4975 raft_consensus.cc:740] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff23af135c614f90bd96dbce69363847, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.992457 4975 consensus_queue.cc:260] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.992538 5018 ts_tablet_manager.cc:1929] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:30.992622 5018 log.cc:1199] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/4501fc26d912471080c3bda747517327
I20260202 15:03:30.992758 4975 ts_tablet_manager.cc:1434] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:03:30.992920 5018 ts_tablet_manager.cc:1950] T 4501fc26d912471080c3bda747517327 P ff23af135c614f90bd96dbce69363847: Deleting consensus metadata
I20260202 15:03:30.994051 4450 catalog_manager.cc:4985] TS ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013): tablet 4501fc26d912471080c3bda747517327 (table test-table [id=bf4b2ecf019f4e14bc0edfc2d702157f]) successfully deleted
I20260202 15:03:30.994794 4977 tablet_bootstrap.cc:492] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: No bootstrap required, opened a new log
I20260202 15:03:30.994913 4977 ts_tablet_manager.cc:1403] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260202 15:03:30.995441 4977 raft_consensus.cc:359] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.995556 4977 raft_consensus.cc:385] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:30.995596 4977 raft_consensus.cc:740] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c7b4cef7974153b7c0290b29e6733f, State: Initialized, Role: FOLLOWER
I20260202 15:03:30.995738 4977 consensus_queue.cc:260] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:30.996032 4977 ts_tablet_manager.cc:1434] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260202 15:03:31.113404 4981 raft_consensus.cc:493] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:31.113561 4981 raft_consensus.cc:515] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:31.114779 4981 leader_election.cc:290] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:31.118121 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.115030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43581 (local address 127.2.164.66:57005)
0202 15:03:31.115377 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:31.115393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:31.115491 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:31.115931 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:31.115934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:31.115948 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:31.116145 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.116153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.116945 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.116950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:31.117682 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.117691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.117827 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.117851 (+ 24us) client_negotiation.cc:769] Sending connection context
0202 15:03:31.117905 (+ 54us) client_negotiation.cc:240] Negotiation successful
0202 15:03:31.117964 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260202 15:03:31.118606 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.115415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:57005 (local address 127.2.164.67:43581)
0202 15:03:31.115649 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:31.115653 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:31.115668 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:03:31.115727 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:31.115730 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:31.115781 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:31.115879 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:31.116285 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.116820 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.117818 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.118312 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.118350 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:31.118402 (+ 52us) server_negotiation.cc:300] Negotiation successful
0202 15:03:31.118463 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
I20260202 15:03:31.119040 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb4c2ec46754fbb8c2e65face473c4f" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f" is_pre_election: true
I20260202 15:03:31.119123 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.114957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44181 (local address 127.2.164.66:34225)
0202 15:03:31.115353 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:31.115368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:31.115497 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:31.115810 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:31.115813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:31.115831 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:31.116119 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.116127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.117183 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.117187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:31.118097 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.118106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.118820 (+ 714us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.118836 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:03:31.118926 (+ 90us) client_negotiation.cc:240] Negotiation successful
0202 15:03:31.118982 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"spinlock_wait_cycles":5376,"thread_start_us":78,"threads_started":1}
I20260202 15:03:31.119161 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.115090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:34225 (local address 127.2.164.65:44181)
0202 15:03:31.115234 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:31.115238 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:31.115439 (+ 201us) server_negotiation.cc:408] Connection header received
0202 15:03:31.115653 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:31.115655 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:31.115707 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:31.115797 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:31.116249 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.117048 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.118242 (+ 1194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.118723 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.118759 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:31.118977 (+ 218us) server_negotiation.cc:300] Negotiation successful
0202 15:03:31.119053 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260202 15:03:31.119194 4903 raft_consensus.cc:2468] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 0.
I20260202 15:03:31.119724 4622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb4c2ec46754fbb8c2e65face473c4f" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e2ba39b9944b55b82a546da8748d88" is_pre_election: true
I20260202 15:03:31.119722 4697 leader_election.cc:304] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c7b4cef7974153b7c0290b29e6733f, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:31.119933 4622 raft_consensus.cc:2468] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 0.
I20260202 15:03:31.119980 4981 raft_consensus.cc:2804] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:31.120028 4981 raft_consensus.cc:493] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:31.120071 4981 raft_consensus.cc:3060] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:31.120947 4981 raft_consensus.cc:515] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:31.121418 4981 leader_election.cc:290] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 election: Requested vote from peers 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:31.121887 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb4c2ec46754fbb8c2e65face473c4f" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f"
I20260202 15:03:31.122013 4903 raft_consensus.cc:3060] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:31.122053 4622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb4c2ec46754fbb8c2e65face473c4f" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e2ba39b9944b55b82a546da8748d88"
I20260202 15:03:31.122171 4622 raft_consensus.cc:3060] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:31.122884 4903 raft_consensus.cc:2468] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 1.
I20260202 15:03:31.123234 4622 raft_consensus.cc:2468] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 1.
I20260202 15:03:31.123242 4697 leader_election.cc:304] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c7b4cef7974153b7c0290b29e6733f, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:31.123432 4981 raft_consensus.cc:2804] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:31.123777 4981 raft_consensus.cc:697] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 1 LEADER]: Becoming Leader. State: Replica: ff23af135c614f90bd96dbce69363847, State: Running, Role: LEADER
I20260202 15:03:31.124219 4981 consensus_queue.cc:237] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:31.127252 4460 catalog_manager.cc:5654] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff23af135c614f90bd96dbce69363847 (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ff23af135c614f90bd96dbce69363847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } health_report { overall_health: UNKNOWN } } }
I20260202 15:03:31.155964 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.152562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.2.164.66:41013)
0202 15:03:31.152901 (+ 339us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:31.152905 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:31.152919 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:03:31.153056 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:31.153060 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:31.153128 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:31.153253 (+ 125us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:31.153658 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.154180 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.154804 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.154986 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.155038 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:31.155511 (+ 473us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:31.155653 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:31.155736 (+ 83us) server_negotiation.cc:300] Negotiation successful
0202 15:03:31.155786 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260202 15:03:31.167155 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.163435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46893 (local address 127.2.164.65:44181)
0202 15:03:31.163560 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:31.163571 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:31.163762 (+ 191us) server_negotiation.cc:408] Connection header received
0202 15:03:31.163893 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:31.163896 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:31.163955 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:31.164042 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:31.164510 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.165308 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.166450 (+ 1142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.166643 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.166689 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:31.166771 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:31.166845 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:31.166982 (+ 137us) server_negotiation.cc:300] Negotiation successful
0202 15:03:31.167031 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260202 15:03:31.167171 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.163335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44181 (local address 127.2.164.66:46893)
0202 15:03:31.163668 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:31.163683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:31.163790 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:31.164061 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:31.164065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:31.164073 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:03:31.164379 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.164385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.165428 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.165432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:31.166314 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.166325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.166440 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.166484 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:03:31.166875 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:03:31.166880 (+ 5us) client_negotiation.cc:769] Sending connection context
0202 15:03:31.166949 (+ 69us) client_negotiation.cc:240] Negotiation successful
0202 15:03:31.167019 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260202 15:03:31.182878 4903 raft_consensus.cc:1275] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Refusing update from remote peer ff23af135c614f90bd96dbce69363847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:31.183060 4622 raft_consensus.cc:1275] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Refusing update from remote peer ff23af135c614f90bd96dbce69363847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:31.183732 5027 consensus_queue.cc:1048] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:31.183995 4981 consensus_queue.cc:1048] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:31.203080 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.200113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:54151 (local address 127.2.164.66:41013)
0202 15:03:31.200341 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:31.200348 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:31.200362 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:03:31.200402 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:31.200405 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:31.200465 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:31.200538 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:31.200959 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.201604 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.202307 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.202524 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.202578 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:31.202673 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:31.202758 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:31.202891 (+ 133us) server_negotiation.cc:300] Negotiation successful
0202 15:03:31.202950 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260202 15:03:31.203073 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:31.199910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41013 (local address 127.2.164.65:54151)
0202 15:03:31.200085 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:31.200101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:31.200182 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:31.200548 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:31.200552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:31.200560 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:03:31.200819 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.200831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.201719 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:31.201724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:31.202179 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:31.202188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:31.202307 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:31.202340 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:03:31.202780 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:03:31.202788 (+ 8us) client_negotiation.cc:769] Sending connection context
0202 15:03:31.202860 (+ 72us) client_negotiation.cc:240] Negotiation successful
0202 15:03:31.202939 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
W20260202 15:03:31.204180 5002 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260202 15:03:32.768148 4460 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:32.768409 4460 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:32.772054 4460 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Sending DeleteTablet for 3 replicas of tablet 3bb4c2ec46754fbb8c2e65face473c4f
I20260202 15:03:32.772873 4886 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bb4c2ec46754fbb8c2e65face473c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:54624
I20260202 15:03:32.772897 4744 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bb4c2ec46754fbb8c2e65face473c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47958
I20260202 15:03:32.773089 4602 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bb4c2ec46754fbb8c2e65face473c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60102
I20260202 15:03:32.773273 5060 tablet_replica.cc:333] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: stopping tablet replica
I20260202 15:03:32.773411 5060 raft_consensus.cc:2243] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:32.773555 5060 raft_consensus.cc:2272] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:32.773723 5062 tablet_replica.cc:333] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: stopping tablet replica
I20260202 15:03:32.773813 5061 tablet_replica.cc:333] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: stopping tablet replica
I20260202 15:03:32.773981 5061 raft_consensus.cc:2243] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:32.774160 5062 raft_consensus.cc:2243] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:32.774292 5060 ts_tablet_manager.cc:1916] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:32.774461 5062 raft_consensus.cc:2272] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:32.774781 5061 raft_consensus.cc:2272] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:32.775473 5062 ts_tablet_manager.cc:1916] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:32.776183 5061 ts_tablet_manager.cc:1916] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:32.777122 5060 ts_tablet_manager.cc:1929] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260202 15:03:32.777201 5060 log.cc:1199] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/3bb4c2ec46754fbb8c2e65face473c4f
I20260202 15:03:32.777484 5060 ts_tablet_manager.cc:1950] T 3bb4c2ec46754fbb8c2e65face473c4f P b5c7b4cef7974153b7c0290b29e6733f: Deleting consensus metadata
I20260202 15:03:32.778295 4449 catalog_manager.cc:4985] TS b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581): tablet 3bb4c2ec46754fbb8c2e65face473c4f (table test-table [id=9d95dc62c92c4ba48d60874ae5b219f4]) successfully deleted
I20260202 15:03:32.778894 5062 ts_tablet_manager.cc:1929] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260202 15:03:32.778890 5061 ts_tablet_manager.cc:1929] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260202 15:03:32.778971 5061 log.cc:1199] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/3bb4c2ec46754fbb8c2e65face473c4f
I20260202 15:03:32.778973 5062 log.cc:1199] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/3bb4c2ec46754fbb8c2e65face473c4f
I20260202 15:03:32.779295 5062 ts_tablet_manager.cc:1950] T 3bb4c2ec46754fbb8c2e65face473c4f P 03e2ba39b9944b55b82a546da8748d88: Deleting consensus metadata
I20260202 15:03:32.779299 5061 ts_tablet_manager.cc:1950] T 3bb4c2ec46754fbb8c2e65face473c4f P ff23af135c614f90bd96dbce69363847: Deleting consensus metadata
I20260202 15:03:32.780123 4450 catalog_manager.cc:4985] TS ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013): tablet 3bb4c2ec46754fbb8c2e65face473c4f (table test-table [id=9d95dc62c92c4ba48d60874ae5b219f4]) successfully deleted
I20260202 15:03:32.780303 4450 catalog_manager.cc:4985] TS 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181): tablet 3bb4c2ec46754fbb8c2e65face473c4f (table test-table [id=9d95dc62c92c4ba48d60874ae5b219f4]) successfully deleted
Feb 02 15:03:32 dist-test-slave-6zzp krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 02 15:03:32 dist-test-slave-6zzp krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:32.803782 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:32.792511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.2.164.126:37519)
0202 15:03:32.792789 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:32.792792 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:32.792808 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:03:32.792929 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:32.792931 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:32.792977 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:32.793137 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:32.794022 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:32.794546 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:32.795153 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:32.795281 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:32.800856 (+ 5575us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:32.800878 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:32.800880 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:32.800906 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:32.802399 (+ 1493us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:32.802749 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:32.802751 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:32.802753 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:32.802791 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:32.803021 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:32.803023 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:32.803024 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:32.803163 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:32.803258 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:32.803520 (+ 262us) server_negotiation.cc:300] Negotiation successful
0202 15:03:32.803637 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":122,"threads_started":1}
I20260202 15:03:32.806871 4460 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:32.807282 4460 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:32.812498 4602 tablet_service.cc:1505] Processing CreateTablet for tablet 142092fa3cba4eafb305aaadc439be46 (DEFAULT_TABLE table=test-table [id=231d821d6f9546128004323136c7af86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:32.812697 4744 tablet_service.cc:1505] Processing CreateTablet for tablet 142092fa3cba4eafb305aaadc439be46 (DEFAULT_TABLE table=test-table [id=231d821d6f9546128004323136c7af86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:32.812800 4602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 142092fa3cba4eafb305aaadc439be46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:32.812903 4886 tablet_service.cc:1505] Processing CreateTablet for tablet 142092fa3cba4eafb305aaadc439be46 (DEFAULT_TABLE table=test-table [id=231d821d6f9546128004323136c7af86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:32.813031 4744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 142092fa3cba4eafb305aaadc439be46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:32.813210 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 142092fa3cba4eafb305aaadc439be46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:32.815147 5071 tablet_bootstrap.cc:492] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Bootstrap starting.
I20260202 15:03:32.815359 5073 tablet_bootstrap.cc:492] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Bootstrap starting.
I20260202 15:03:32.816040 5071 tablet_bootstrap.cc:654] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:32.816226 5073 tablet_bootstrap.cc:654] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:32.816789 5072 tablet_bootstrap.cc:492] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Bootstrap starting.
I20260202 15:03:32.817237 5071 tablet_bootstrap.cc:492] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: No bootstrap required, opened a new log
I20260202 15:03:32.817310 5071 ts_tablet_manager.cc:1403] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:32.817764 5072 tablet_bootstrap.cc:654] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:32.817730 5071 raft_consensus.cc:359] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.817811 5071 raft_consensus.cc:385] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:32.817833 5071 raft_consensus.cc:740] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c7b4cef7974153b7c0290b29e6733f, State: Initialized, Role: FOLLOWER
I20260202 15:03:32.817967 5071 consensus_queue.cc:260] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.818344 5071 ts_tablet_manager.cc:1434] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:03:32.819011 5072 tablet_bootstrap.cc:492] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: No bootstrap required, opened a new log
I20260202 15:03:32.819093 5073 tablet_bootstrap.cc:492] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: No bootstrap required, opened a new log
I20260202 15:03:32.819118 5072 ts_tablet_manager.cc:1403] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:32.819186 5073 ts_tablet_manager.cc:1403] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260202 15:03:32.819504 5072 raft_consensus.cc:359] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.819613 5072 raft_consensus.cc:385] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:32.819664 5072 raft_consensus.cc:740] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff23af135c614f90bd96dbce69363847, State: Initialized, Role: FOLLOWER
I20260202 15:03:32.819655 5073 raft_consensus.cc:359] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.819793 5073 raft_consensus.cc:385] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:32.819844 5073 raft_consensus.cc:740] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Initialized, Role: FOLLOWER
I20260202 15:03:32.819905 5072 consensus_queue.cc:260] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.820173 5072 ts_tablet_manager.cc:1434] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:32.820003 5073 consensus_queue.cc:260] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.820957 5073 ts_tablet_manager.cc:1434] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:32.917412 5041 raft_consensus.cc:493] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:32.917593 5041 raft_consensus.cc:515] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.918076 5041 leader_election.cc:290] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:32.918502 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "142092fa3cba4eafb305aaadc439be46" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f" is_pre_election: true
I20260202 15:03:32.918540 4764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "142092fa3cba4eafb305aaadc439be46" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff23af135c614f90bd96dbce69363847" is_pre_election: true
I20260202 15:03:32.918670 4903 raft_consensus.cc:2468] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 0.
I20260202 15:03:32.918706 4764 raft_consensus.cc:2468] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 0.
I20260202 15:03:32.919056 4556 leader_election.cc:304] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:32.919246 5041 raft_consensus.cc:2804] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:32.919317 5041 raft_consensus.cc:493] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:32.919368 5041 raft_consensus.cc:3060] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:32.920694 5041 raft_consensus.cc:515] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.921136 5041 leader_election.cc:290] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 election: Requested vote from peers ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:32.921589 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "142092fa3cba4eafb305aaadc439be46" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f"
I20260202 15:03:32.921784 4903 raft_consensus.cc:3060] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:32.921891 4764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "142092fa3cba4eafb305aaadc439be46" candidate_uuid: "03e2ba39b9944b55b82a546da8748d88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff23af135c614f90bd96dbce69363847"
I20260202 15:03:32.922039 4764 raft_consensus.cc:3060] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:32.922827 4903 raft_consensus.cc:2468] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 1.
I20260202 15:03:32.923048 4764 raft_consensus.cc:2468] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e2ba39b9944b55b82a546da8748d88 in term 1.
I20260202 15:03:32.923269 4555 leader_election.cc:304] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, b5c7b4cef7974153b7c0290b29e6733f; no voters:
I20260202 15:03:32.923499 5041 raft_consensus.cc:2804] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:32.923599 5041 raft_consensus.cc:697] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 1 LEADER]: Becoming Leader. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Running, Role: LEADER
I20260202 15:03:32.923764 5041 consensus_queue.cc:237] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:32.925499 4461 catalog_manager.cc:5654] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "03e2ba39b9944b55b82a546da8748d88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } health_report { overall_health: UNKNOWN } } }
I20260202 15:03:32.952272 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:32.948765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.2.164.65:44181)
0202 15:03:32.949354 (+ 589us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:32.949359 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:32.949383 (+ 24us) server_negotiation.cc:408] Connection header received
0202 15:03:32.949448 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:32.949451 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:32.949519 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:32.949639 (+ 120us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:32.950037 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:32.950554 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:32.951294 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:32.951502 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:32.951559 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:32.951659 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:32.951860 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:32.951974 (+ 114us) server_negotiation.cc:300] Negotiation successful
0202 15:03:32.952087 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":127,"threads_started":1}
I20260202 15:03:32.958525 4764 raft_consensus.cc:1275] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Refusing update from remote peer 03e2ba39b9944b55b82a546da8748d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:32.958596 4903 raft_consensus.cc:1275] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Refusing update from remote peer 03e2ba39b9944b55b82a546da8748d88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:32.958935 5016 consensus_queue.cc:1048] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:32.959113 5046 consensus_queue.cc:1048] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:32.989423 4463 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:32.989590 4463 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:32.991600 4463 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Sending DeleteTablet for 3 replicas of tablet 142092fa3cba4eafb305aaadc439be46
I20260202 15:03:32.992266 4886 tablet_service.cc:1552] Processing DeleteTablet for tablet 142092fa3cba4eafb305aaadc439be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:54624
I20260202 15:03:32.992319 4602 tablet_service.cc:1552] Processing DeleteTablet for tablet 142092fa3cba4eafb305aaadc439be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60102
I20260202 15:03:32.992342 4744 tablet_service.cc:1552] Processing DeleteTablet for tablet 142092fa3cba4eafb305aaadc439be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47958
I20260202 15:03:32.992483 5062 tablet_replica.cc:333] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: stopping tablet replica
I20260202 15:03:32.992516 5061 tablet_replica.cc:333] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: stopping tablet replica
I20260202 15:03:32.992635 5061 raft_consensus.cc:2243] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:32.992637 5062 raft_consensus.cc:2243] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:32.992676 5060 tablet_replica.cc:333] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: stopping tablet replica
I20260202 15:03:32.992763 5061 raft_consensus.cc:2272] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:32.992789 5060 raft_consensus.cc:2243] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:32.992851 5062 raft_consensus.cc:2272] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:32.992888 5060 raft_consensus.cc:2272] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:32.993160 4463 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:32.993593 4463 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:32.993642 5061 ts_tablet_manager.cc:1916] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:32.993677 5060 ts_tablet_manager.cc:1916] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:32.995239 5062 ts_tablet_manager.cc:1916] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:32.995887 5060 ts_tablet_manager.cc:1929] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:32.995962 5061 ts_tablet_manager.cc:1929] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:32.995981 5060 log.cc:1199] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/142092fa3cba4eafb305aaadc439be46
I20260202 15:03:32.996037 5061 log.cc:1199] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/142092fa3cba4eafb305aaadc439be46
I20260202 15:03:32.996241 5060 ts_tablet_manager.cc:1950] T 142092fa3cba4eafb305aaadc439be46 P b5c7b4cef7974153b7c0290b29e6733f: Deleting consensus metadata
I20260202 15:03:32.996305 5061 ts_tablet_manager.cc:1950] T 142092fa3cba4eafb305aaadc439be46 P ff23af135c614f90bd96dbce69363847: Deleting consensus metadata
I20260202 15:03:32.997040 4449 catalog_manager.cc:4985] TS b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581): tablet 142092fa3cba4eafb305aaadc439be46 (table test-table [id=231d821d6f9546128004323136c7af86]) successfully deleted
I20260202 15:03:32.997234 4450 catalog_manager.cc:4985] TS ff23af135c614f90bd96dbce69363847 (127.2.164.66:41013): tablet 142092fa3cba4eafb305aaadc439be46 (table test-table [id=231d821d6f9546128004323136c7af86]) successfully deleted
I20260202 15:03:32.997769 5062 ts_tablet_manager.cc:1929] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:03:32.997874 5062 log.cc:1199] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/142092fa3cba4eafb305aaadc439be46
I20260202 15:03:32.998117 5062 ts_tablet_manager.cc:1950] T 142092fa3cba4eafb305aaadc439be46 P 03e2ba39b9944b55b82a546da8748d88: Deleting consensus metadata
I20260202 15:03:32.998785 4450 catalog_manager.cc:4985] TS 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181): tablet 142092fa3cba4eafb305aaadc439be46 (table test-table [id=231d821d6f9546128004323136c7af86]) successfully deleted
I20260202 15:03:33.000316 4602 tablet_service.cc:1505] Processing CreateTablet for tablet bf21ce84f1c245d88c3d781b526e05e2 (DEFAULT_TABLE table=test-table [id=e23a323fdffd4564871eafbbc7ce7830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:33.000355 4744 tablet_service.cc:1505] Processing CreateTablet for tablet bf21ce84f1c245d88c3d781b526e05e2 (DEFAULT_TABLE table=test-table [id=e23a323fdffd4564871eafbbc7ce7830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:33.000577 4602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf21ce84f1c245d88c3d781b526e05e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:33.000577 4744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf21ce84f1c245d88c3d781b526e05e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:33.000777 4886 tablet_service.cc:1505] Processing CreateTablet for tablet bf21ce84f1c245d88c3d781b526e05e2 (DEFAULT_TABLE table=test-table [id=e23a323fdffd4564871eafbbc7ce7830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:33.001070 4886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf21ce84f1c245d88c3d781b526e05e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:33.002637 5071 tablet_bootstrap.cc:492] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Bootstrap starting.
I20260202 15:03:33.002714 5073 tablet_bootstrap.cc:492] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88: Bootstrap starting.
I20260202 15:03:33.003468 5072 tablet_bootstrap.cc:492] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Bootstrap starting.
I20260202 15:03:33.003580 5073 tablet_bootstrap.cc:654] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:33.003675 5071 tablet_bootstrap.cc:654] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:33.004446 5072 tablet_bootstrap.cc:654] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:33.004586 5073 tablet_bootstrap.cc:492] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88: No bootstrap required, opened a new log
I20260202 15:03:33.004673 5073 ts_tablet_manager.cc:1403] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:33.005177 5073 raft_consensus.cc:359] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.005273 5073 raft_consensus.cc:385] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:33.005244 5071 tablet_bootstrap.cc:492] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: No bootstrap required, opened a new log
I20260202 15:03:33.005296 5073 raft_consensus.cc:740] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e2ba39b9944b55b82a546da8748d88, State: Initialized, Role: FOLLOWER
I20260202 15:03:33.005347 5071 ts_tablet_manager.cc:1403] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260202 15:03:33.005434 5073 consensus_queue.cc:260] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.005725 5073 ts_tablet_manager.cc:1434] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:33.005867 5071 raft_consensus.cc:359] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.005973 5071 raft_consensus.cc:385] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:33.006011 5071 raft_consensus.cc:740] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c7b4cef7974153b7c0290b29e6733f, State: Initialized, Role: FOLLOWER
I20260202 15:03:33.006039 5072 tablet_bootstrap.cc:492] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: No bootstrap required, opened a new log
I20260202 15:03:33.006109 5072 ts_tablet_manager.cc:1403] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260202 15:03:33.006127 5071 consensus_queue.cc:260] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.006356 5071 ts_tablet_manager.cc:1434] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:33.006500 5072 raft_consensus.cc:359] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.006610 5072 raft_consensus.cc:385] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:33.006644 5072 raft_consensus.cc:740] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff23af135c614f90bd96dbce69363847, State: Initialized, Role: FOLLOWER
I20260202 15:03:33.006759 5072 consensus_queue.cc:260] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.006986 5072 ts_tablet_manager.cc:1434] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:03:33.030100 5051 raft_consensus.cc:493] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:33.030254 5051 raft_consensus.cc:515] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.030735 5051 leader_election.cc:290] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:33.031119 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf21ce84f1c245d88c3d781b526e05e2" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f" is_pre_election: true
I20260202 15:03:33.031273 4903 raft_consensus.cc:2468] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 0.
I20260202 15:03:33.031229 4622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf21ce84f1c245d88c3d781b526e05e2" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e2ba39b9944b55b82a546da8748d88" is_pre_election: true
I20260202 15:03:33.031714 4622 raft_consensus.cc:2468] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 0.
I20260202 15:03:33.031853 4697 leader_election.cc:304] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c7b4cef7974153b7c0290b29e6733f, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:33.032047 5051 raft_consensus.cc:2804] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:33.032096 5051 raft_consensus.cc:493] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:33.032126 5051 raft_consensus.cc:3060] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:33.033178 5051 raft_consensus.cc:515] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.033560 5051 leader_election.cc:290] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 election: Requested vote from peers 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181), b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581)
I20260202 15:03:33.033922 4622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf21ce84f1c245d88c3d781b526e05e2" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e2ba39b9944b55b82a546da8748d88"
I20260202 15:03:33.033977 4903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf21ce84f1c245d88c3d781b526e05e2" candidate_uuid: "ff23af135c614f90bd96dbce69363847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c7b4cef7974153b7c0290b29e6733f"
I20260202 15:03:33.034054 4622 raft_consensus.cc:3060] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:33.034123 4903 raft_consensus.cc:3060] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:33.034960 4622 raft_consensus.cc:2468] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 1.
I20260202 15:03:33.034968 4903 raft_consensus.cc:2468] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff23af135c614f90bd96dbce69363847 in term 1.
I20260202 15:03:33.035310 4698 leader_election.cc:304] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e2ba39b9944b55b82a546da8748d88, ff23af135c614f90bd96dbce69363847; no voters:
I20260202 15:03:33.035501 5051 raft_consensus.cc:2804] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:33.035596 5051 raft_consensus.cc:697] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 1 LEADER]: Becoming Leader. State: Replica: ff23af135c614f90bd96dbce69363847, State: Running, Role: LEADER
I20260202 15:03:33.035754 5051 consensus_queue.cc:237] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } }
I20260202 15:03:33.037479 4463 catalog_manager.cc:5654] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff23af135c614f90bd96dbce69363847 (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ff23af135c614f90bd96dbce69363847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff23af135c614f90bd96dbce69363847" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 } health_report { overall_health: UNKNOWN } } }
I20260202 15:03:33.061335 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:33.058458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.2.164.66:41013)
0202 15:03:33.058756 (+ 298us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:33.058761 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:33.058781 (+ 20us) server_negotiation.cc:408] Connection header received
0202 15:03:33.058815 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:33.058818 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:33.058907 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:33.059017 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:33.059368 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:33.059910 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:33.060543 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:33.060731 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:33.060773 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:33.060839 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:33.060937 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:33.061130 (+ 193us) server_negotiation.cc:300] Negotiation successful
0202 15:03:33.061180 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260202 15:03:33.067875 4906 raft_consensus.cc:1275] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Refusing update from remote peer ff23af135c614f90bd96dbce69363847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:33.067981 4622 raft_consensus.cc:1275] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Refusing update from remote peer ff23af135c614f90bd96dbce69363847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:33.068301 5055 consensus_queue.cc:1048] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c7b4cef7974153b7c0290b29e6733f" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:33.068490 5051 consensus_queue.cc:1048] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e2ba39b9944b55b82a546da8748d88" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:03:33.083554 5042 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260202 15:03:34.633414 4460 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:34.633558 4460 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:34.635833 4460 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 092aae1ff77b450387e069d2f2e1ddc2: Sending DeleteTablet for 3 replicas of tablet bf21ce84f1c245d88c3d781b526e05e2
I20260202 15:03:34.636405 4602 tablet_service.cc:1552] Processing DeleteTablet for tablet bf21ce84f1c245d88c3d781b526e05e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:34 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60102
I20260202 15:03:34.636514 4886 tablet_service.cc:1552] Processing DeleteTablet for tablet bf21ce84f1c245d88c3d781b526e05e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:34 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:54624
I20260202 15:03:34.636518 4744 tablet_service.cc:1552] Processing DeleteTablet for tablet bf21ce84f1c245d88c3d781b526e05e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:34 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47958
I20260202 15:03:34.636874 5114 tablet_replica.cc:333] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: stopping tablet replica
I20260202 15:03:34.636894 5113 tablet_replica.cc:333] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88: stopping tablet replica
I20260202 15:03:34.637046 5113 raft_consensus.cc:2243] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:34.637181 5113 raft_consensus.cc:2272] T bf21ce84f1c245d88c3d781b526e05e2 P 03e2ba39b9944b55b82a546da8748d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:34.637192 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 4537
I20260202 15:03:34.638962 5115 tablet_replica.cc:333] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: stopping tablet replica
I20260202 15:03:34.639098 5115 raft_consensus.cc:2243] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:34.639231 5115 raft_consensus.cc:2272] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:34.639433 5114 raft_consensus.cc:2243] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:34.639647 5114 raft_consensus.cc:2272] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:34.640182 5115 ts_tablet_manager.cc:1916] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:34.642328 5115 ts_tablet_manager.cc:1929] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260202 15:03:34.642423 5115 log.cc:1199] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/bf21ce84f1c245d88c3d781b526e05e2
I20260202 15:03:34.642737 5115 ts_tablet_manager.cc:1950] T bf21ce84f1c245d88c3d781b526e05e2 P b5c7b4cef7974153b7c0290b29e6733f: Deleting consensus metadata
I20260202 15:03:34.643041 5114 ts_tablet_manager.cc:1916] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:34.643515 4449 catalog_manager.cc:4985] TS b5c7b4cef7974153b7c0290b29e6733f (127.2.164.67:43581): tablet bf21ce84f1c245d88c3d781b526e05e2 (table test-table [id=e23a323fdffd4564871eafbbc7ce7830]) successfully deleted
W20260202 15:03:34.645545 4450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:44181 (error 108)
I20260202 15:03:34.646188 5114 ts_tablet_manager.cc:1929] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260202 15:03:34.646281 5114 log.cc:1199] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/bf21ce84f1c245d88c3d781b526e05e2
I20260202 15:03:34.646592 5114 ts_tablet_manager.cc:1950] T bf21ce84f1c245d88c3d781b526e05e2 P ff23af135c614f90bd96dbce69363847: Deleting consensus metadata
I20260202 15:03:34.646839 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:34.646365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44181 (local address 127.0.0.1:60144)
0202 15:03:34.646605 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:34.646681 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260202 15:03:34.646873 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 4679
W20260202 15:03:34.654003 4450 catalog_manager.cc:4712] TS 03e2ba39b9944b55b82a546da8748d88 (127.2.164.65:44181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf21ce84f1c245d88c3d781b526e05e2: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44181: connect: Connection refused (error 111)
I20260202 15:03:34.654161 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 4821
I20260202 15:03:34.655450 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:34.655095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41013 (local address 127.0.0.1:47978)
0202 15:03:34.655277 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:34.655338 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.66:41013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260202 15:03:34.660375 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 4431
2026-02-02T15:03:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7396 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:34 dist-test-slave-6zzp krb5kdc[5120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:34 dist-test-slave-6zzp krb5kdc[5120](info): set up 2 sockets
Feb 02 15:03:34 dist-test-slave-6zzp krb5kdc[5120](info): commencing operation
krb5kdc: starting...
W20260202 15:03:36.663389 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260202 15:03:36.882804 2705 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.219s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260202 15:03:37.164561 2705 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.282s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260202 15:03:37.453213 2705 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.288s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/test-user.keytab.
W20260202 15:03:38.458066 2705 mini_kdc.cc:290] Time spent creating keytab for test-user: real 1.005s user 0.000s sys 0.002s
Feb 02 15:03:38 dist-test-slave-6zzp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
W20260202 15:03:39.577728 2705 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.2.164.126: real 1.097s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:39.600383 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34679
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:39.715785 5136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:39.716060 5136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:39.716120 5136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:39.719571 5136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:39.719648 5136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:39.719673 5136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:39.719693 5136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:39.719710 5136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:39.724236 5136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:39.725476 5136 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:39.726384 5136 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260202 15:03:39.732708 5136 server_base.cc:1060] running on GCE node
W20260202 15:03:39.732725 5142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:39.732941 5141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:39.733042 5144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:39.733577 5136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:39.734568 5136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:39.735726 5136 hybrid_clock.cc:648] HybridClock initialized: now 1770044619735696 us; error 46 us; skew 500 ppm
Feb 02 15:03:39 dist-test-slave-6zzp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:39.738512 5136 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:39.739681 5136 webserver.cc:492] Webserver started at http://127.2.164.126:44455/ using document root <none> and password file <none>
I20260202 15:03:39.740298 5136 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:39.740350 5136 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:39.740569 5136 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:39.742436 5136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "b50dc5f678ba4a10bd9bc767c658103a"
format_stamp: "Formatted at 2026-02-02 15:03:39 on dist-test-slave-6zzp"
server_key: "00a5d3c1ebec919fc87d6253a841ad0a"
server_key_iv: "7e37765df644e1c6cee812d30bb30628"
server_key_version: "encryptionkey@0"
I20260202 15:03:39.742950 5136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "b50dc5f678ba4a10bd9bc767c658103a"
format_stamp: "Formatted at 2026-02-02 15:03:39 on dist-test-slave-6zzp"
server_key: "00a5d3c1ebec919fc87d6253a841ad0a"
server_key_iv: "7e37765df644e1c6cee812d30bb30628"
server_key_version: "encryptionkey@0"
I20260202 15:03:39.746687 5136 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260202 15:03:39.749126 5151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:39.750396 5136 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260202 15:03:39.750550 5136 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "b50dc5f678ba4a10bd9bc767c658103a"
format_stamp: "Formatted at 2026-02-02 15:03:39 on dist-test-slave-6zzp"
server_key: "00a5d3c1ebec919fc87d6253a841ad0a"
server_key_iv: "7e37765df644e1c6cee812d30bb30628"
server_key_version: "encryptionkey@0"
I20260202 15:03:39.750669 5136 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:39.776957 5136 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:39.785200 5136 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:39.785470 5136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:39.794545 5203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:34679 every 8 connection(s)
I20260202 15:03:39.794569 5136 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:34679
I20260202 15:03:39.796288 5136 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:39.796835 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5136
I20260202 15:03:39.796958 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:39.797328 2705 external_mini_cluster.cc:1468] Setting key 2a8ff9ebc1c6bbb5e2574879826b8720
I20260202 15:03:39.800953 5204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Feb 02 15:03:39 dist-test-slave-6zzp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:39.808095 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a: Bootstrap starting.
I20260202 15:03:39.810812 5204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:39.811606 5204 log.cc:826] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:39.811638 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:39.798950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.2.164.126:34679)
0202 15:03:39.799432 (+ 482us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:39.799444 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:39.799484 (+ 40us) server_negotiation.cc:408] Connection header received
0202 15:03:39.800225 (+ 741us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:39.800246 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:39.800603 (+ 357us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:39.800932 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:39.802219 (+ 1287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:39.803021 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:39.803888 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:39.804158 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:39.806870 (+ 2712us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:39.806923 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:39.806935 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:39.806977 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:39.808770 (+ 1793us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:39.809315 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:39.809320 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:39.809339 (+ 19us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:39.809424 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:39.809667 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:39.809669 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:39.809670 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:39.810087 (+ 417us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:39.810282 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:39.810616 (+ 334us) server_negotiation.cc:300] Negotiation successful
0202 15:03:39.810971 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":161,"threads_started":1}
I20260202 15:03:39.813900 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a: No bootstrap required, opened a new log
I20260202 15:03:39.816498 5204 raft_consensus.cc:359] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50dc5f678ba4a10bd9bc767c658103a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34679 } }
I20260202 15:03:39.816710 5204 raft_consensus.cc:385] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:39.816754 5204 raft_consensus.cc:740] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50dc5f678ba4a10bd9bc767c658103a, State: Initialized, Role: FOLLOWER
I20260202 15:03:39.817306 5204 consensus_queue.cc:260] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50dc5f678ba4a10bd9bc767c658103a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34679 } }
I20260202 15:03:39.817474 5204 raft_consensus.cc:399] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:39.817550 5204 raft_consensus.cc:493] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:39.817658 5204 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:39.818650 5204 raft_consensus.cc:515] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50dc5f678ba4a10bd9bc767c658103a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34679 } }
I20260202 15:03:39.819026 5204 leader_election.cc:304] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b50dc5f678ba4a10bd9bc767c658103a; no voters:
I20260202 15:03:39.819350 5204 leader_election.cc:290] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:39.819522 5209 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:39.819762 5209 raft_consensus.cc:697] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [term 1 LEADER]: Becoming Leader. State: Replica: b50dc5f678ba4a10bd9bc767c658103a, State: Running, Role: LEADER
I20260202 15:03:39.820077 5209 consensus_queue.cc:237] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50dc5f678ba4a10bd9bc767c658103a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34679 } }
I20260202 15:03:39.820562 5204 sys_catalog.cc:565] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:39.821924 5211 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b50dc5f678ba4a10bd9bc767c658103a. Latest consensus state: current_term: 1 leader_uuid: "b50dc5f678ba4a10bd9bc767c658103a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50dc5f678ba4a10bd9bc767c658103a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34679 } } }
I20260202 15:03:39.822080 5211 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:39.822400 5210 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b50dc5f678ba4a10bd9bc767c658103a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50dc5f678ba4a10bd9bc767c658103a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34679 } } }
I20260202 15:03:39.822549 5210 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:39.822467 5218 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:39.826227 5218 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:39.831625 5218 catalog_manager.cc:1357] Generated new cluster ID: b58b761aa87b4620a7fb9843427f406c
I20260202 15:03:39.831709 5218 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:39.853281 5218 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:39.854265 5218 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:39.863587 5218 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b50dc5f678ba4a10bd9bc767c658103a: Generated new TSK 0
I20260202 15:03:39.864375 5218 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:39.922389 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34679
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:40.031924 5232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:40.032145 5232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:40.032202 5232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:40.035517 5232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:40.035578 5232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:40.035701 5232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:40.039985 5232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:34679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:40.041108 5232 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:40.041935 5232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:40.048394 5240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:40.048486 5238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:40.048399 5237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:40.048606 5232 server_base.cc:1060] running on GCE node
I20260202 15:03:40.049072 5232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:40.049602 5232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:40.050770 5232 hybrid_clock.cc:648] HybridClock initialized: now 1770044620050752 us; error 45 us; skew 500 ppm
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:40.053256 5232 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:40.054282 5232 webserver.cc:492] Webserver started at http://127.2.164.65:41081/ using document root <none> and password file <none>
I20260202 15:03:40.054865 5232 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:40.054940 5232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:40.055153 5232 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:40.057041 5232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "036e67300af2410c8f4971d427b7ff78"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "5cea5d6a95f26743d3f798129b76d87f"
server_key_iv: "8833021687ab540b5cdeb083f478e2d2"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.057578 5232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "036e67300af2410c8f4971d427b7ff78"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "5cea5d6a95f26743d3f798129b76d87f"
server_key_iv: "8833021687ab540b5cdeb083f478e2d2"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.061105 5232 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:03:40.063333 5247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.064493 5232 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:40.064625 5232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "036e67300af2410c8f4971d427b7ff78"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "5cea5d6a95f26743d3f798129b76d87f"
server_key_iv: "8833021687ab540b5cdeb083f478e2d2"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.064731 5232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:40.082022 5232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:40.084931 5232 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:40.085176 5232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:40.085767 5232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:40.086728 5232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:40.086802 5232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.086876 5232 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:40.086930 5232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.096124 5232 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:38835
I20260202 15:03:40.096154 5360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:38835 every 8 connection(s)
I20260202 15:03:40.097250 5232 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:40.098323 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5232
I20260202 15:03:40.098436 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:40.098697 2705 external_mini_cluster.cc:1468] Setting key 76c07740bfd84d69f9ddb238b15cf255
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:40.111039 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:40.099243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:57201 (local address 127.2.164.126:34679)
0202 15:03:40.099414 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:40.099418 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:40.100171 (+ 753us) server_negotiation.cc:408] Connection header received
0202 15:03:40.101254 (+ 1083us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:40.101260 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:40.101321 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:40.101408 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:40.103044 (+ 1636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.103733 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:40.104693 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.104856 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:40.107342 (+ 2486us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:40.107368 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:40.107371 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:40.107395 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:40.108912 (+ 1517us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:40.109456 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:40.109459 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:40.109460 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:40.109501 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:40.109905 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:40.109908 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:40.109910 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:40.110063 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:40.110147 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:40.110696 (+ 549us) server_negotiation.cc:300] Negotiation successful
0202 15:03:40.110828 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:03:40.111641 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:40.099539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34679 (local address 127.2.164.65:57201)
0202 15:03:40.100027 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:40.100082 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:40.101016 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:40.101549 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:40.101557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:40.101952 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:40.102909 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:40.102923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.103889 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:40.103896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:40.104557 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:40.104571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.104823 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:40.105466 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:40.105512 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:40.107195 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:40.109069 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:40.109075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:40.109096 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:40.109325 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:40.109582 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:40.109584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:40.109586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:40.109672 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:40.110155 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:40.110160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:40.110422 (+ 262us) client_negotiation.cc:769] Sending connection context
0202 15:03:40.110606 (+ 184us) client_negotiation.cc:240] Negotiation successful
0202 15:03:40.110821 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260202 15:03:40.112993 5361 heartbeater.cc:344] Connected to a master server at 127.2.164.126:34679
I20260202 15:03:40.113313 5361 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:40.113929 5361 heartbeater.cc:507] Master 127.2.164.126:34679 requested a full tablet report, sending...
I20260202 15:03:40.115438 5168 ts_manager.cc:194] Registered new tserver with Master: 036e67300af2410c8f4971d427b7ff78 (127.2.164.65:38835)
I20260202 15:03:40.116631 5168 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:57201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:40.153162 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34679
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:40.261970 5368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:40.262218 5368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:40.262307 5368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:40.265630 5368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:40.265717 5368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:40.265843 5368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:40.270118 5368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:34679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:40.271270 5368 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:40.272126 5368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:40.278752 5376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:40.278721 5373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:40.278721 5374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:40.279161 5368 server_base.cc:1060] running on GCE node
I20260202 15:03:40.279521 5368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:40.280067 5368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:40.281212 5368 hybrid_clock.cc:648] HybridClock initialized: now 1770044620281190 us; error 36 us; skew 500 ppm
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:40.284103 5368 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:40.285244 5368 webserver.cc:492] Webserver started at http://127.2.164.66:45065/ using document root <none> and password file <none>
I20260202 15:03:40.285823 5368 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:40.285895 5368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:40.286121 5368 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:40.287915 5368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "47c4ac6008f14ee590440e04273cbd80"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "93a2f8851c6a0a0a2f62659ba0047177"
server_key_iv: "1983be44f62794de27879b1850a9ca7d"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.288425 5368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "47c4ac6008f14ee590440e04273cbd80"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "93a2f8851c6a0a0a2f62659ba0047177"
server_key_iv: "1983be44f62794de27879b1850a9ca7d"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.291924 5368 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260202 15:03:40.294155 5383 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.295152 5368 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:40.295275 5368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "47c4ac6008f14ee590440e04273cbd80"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "93a2f8851c6a0a0a2f62659ba0047177"
server_key_iv: "1983be44f62794de27879b1850a9ca7d"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.295373 5368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:40.306555 5368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:40.309063 5368 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:40.309257 5368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:40.309788 5368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:40.310632 5368 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:40.310703 5368 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.310771 5368 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:40.310813 5368 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.319339 5368 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:36927
I20260202 15:03:40.319360 5496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:36927 every 8 connection(s)
I20260202 15:03:40.320238 5368 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:40.329105 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5368
I20260202 15:03:40.329231 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:40.329483 2705 external_mini_cluster.cc:1468] Setting key b988d2af3640202005484fb18a2e5b5d
I20260202 15:03:40.333207 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:40.322031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35715 (local address 127.2.164.126:34679)
0202 15:03:40.322202 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:40.322207 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:40.322966 (+ 759us) server_negotiation.cc:408] Connection header received
0202 15:03:40.323778 (+ 812us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:40.323781 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:40.323831 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:40.323911 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:40.325676 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.326194 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:40.326866 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.327024 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:40.329230 (+ 2206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:40.329255 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:40.329258 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:40.329301 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:40.330806 (+ 1505us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:40.331402 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:40.331405 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:40.331407 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:40.331463 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:40.331841 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:40.331844 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:40.331846 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:40.332016 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:40.332102 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:40.332814 (+ 712us) server_negotiation.cc:300] Negotiation successful
0202 15:03:40.332966 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260202 15:03:40.333704 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:40.322322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34679 (local address 127.2.164.66:35715)
0202 15:03:40.322825 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:40.322873 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:40.323601 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:40.324095 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:40.324104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:40.324494 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:40.325516 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:40.325531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.326324 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:40.326328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:40.326733 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:40.326740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.326918 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:40.327518 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:40.327543 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:40.329058 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:40.330917 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:40.330922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:40.330934 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:40.331285 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:40.331562 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:40.331568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:40.331573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:40.331722 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:40.332122 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:40.332132 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:40.332468 (+ 336us) client_negotiation.cc:769] Sending connection context
0202 15:03:40.332701 (+ 233us) client_negotiation.cc:240] Negotiation successful
0202 15:03:40.332938 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260202 15:03:40.334970 5497 heartbeater.cc:344] Connected to a master server at 127.2.164.126:34679
I20260202 15:03:40.335251 5497 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:40.335803 5497 heartbeater.cc:507] Master 127.2.164.126:34679 requested a full tablet report, sending...
I20260202 15:03:40.336947 5168 ts_manager.cc:194] Registered new tserver with Master: 47c4ac6008f14ee590440e04273cbd80 (127.2.164.66:36927)
I20260202 15:03:40.337739 5168 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:35715
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:40.382071 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34679
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:40.492332 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:40.492583 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:40.492671 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:40.495922 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:40.496021 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:40.496133 5504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:40.500622 5504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:34679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:40.501786 5504 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:40.502678 5504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:40.509444 5509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:40.509456 5510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:40.509439 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:40.509797 5504 server_base.cc:1060] running on GCE node
I20260202 15:03:40.510152 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:40.510695 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:40.511845 5504 hybrid_clock.cc:648] HybridClock initialized: now 1770044620511830 us; error 30 us; skew 500 ppm
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:40.514617 5504 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:40.515668 5504 webserver.cc:492] Webserver started at http://127.2.164.67:38855/ using document root <none> and password file <none>
I20260202 15:03:40.516234 5504 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:40.516285 5504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:40.516481 5504 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:40.518472 5504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "212239e7ef75440cb8d9fcaf9fc0ec55"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "41a40fcc2bd2bd15739bc3cbd47a1c44"
server_key_iv: "d73312357b567059a560012222ea0141"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.519083 5504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "212239e7ef75440cb8d9fcaf9fc0ec55"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "41a40fcc2bd2bd15739bc3cbd47a1c44"
server_key_iv: "d73312357b567059a560012222ea0141"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.522764 5504 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260202 15:03:40.525166 5519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.526387 5504 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:40.526542 5504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "212239e7ef75440cb8d9fcaf9fc0ec55"
format_stamp: "Formatted at 2026-02-02 15:03:40 on dist-test-slave-6zzp"
server_key: "41a40fcc2bd2bd15739bc3cbd47a1c44"
server_key_iv: "d73312357b567059a560012222ea0141"
server_key_version: "encryptionkey@0"
I20260202 15:03:40.526649 5504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:40.544854 5504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:40.548702 5504 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:40.548931 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:40.549638 5504 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:40.550488 5504 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:40.550539 5504 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.550578 5504 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:40.550593 5504 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:40.560240 5504 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:40805
I20260202 15:03:40.560257 5632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:40805 every 8 connection(s)
I20260202 15:03:40.561286 5504 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:40.568285 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5504
I20260202 15:03:40.568449 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:40.568698 2705 external_mini_cluster.cc:1468] Setting key 6b8e25e601f8973f59b1e9e1fe50366e
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:40.573642 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:40.563020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59515 (local address 127.2.164.126:34679)
0202 15:03:40.563183 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:40.563189 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:40.563880 (+ 691us) server_negotiation.cc:408] Connection header received
0202 15:03:40.564659 (+ 779us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:40.564661 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:40.564709 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:40.564784 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:40.566461 (+ 1677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.566980 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:40.567586 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.567710 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:40.570252 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:40.570268 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:40.570272 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:40.570298 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:40.571678 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:40.572154 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:40.572158 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:40.572160 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:40.572207 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:40.572483 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:40.572487 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:40.572489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:40.572648 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:40.572748 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:40.573345 (+ 597us) server_negotiation.cc:300] Negotiation successful
0202 15:03:40.573472 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:03:40.574250 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:40.563329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34679 (local address 127.2.164.67:59515)
0202 15:03:40.563733 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:40.563768 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:40.564487 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:40.564931 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:40.564938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:40.565382 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:40.566323 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:40.566337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.567080 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:40.567083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:40.567493 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:40.567500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:40.567670 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:40.568304 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:40.568328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:40.570107 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:40.571795 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:40.571801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:40.571812 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:40.572048 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:40.572294 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:40.572298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:40.572300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:40.572388 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:40.572749 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:40.572758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:40.573056 (+ 298us) client_negotiation.cc:769] Sending connection context
0202 15:03:40.573297 (+ 241us) client_negotiation.cc:240] Negotiation successful
0202 15:03:40.573564 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260202 15:03:40.575376 5633 heartbeater.cc:344] Connected to a master server at 127.2.164.126:34679
I20260202 15:03:40.575595 5633 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:40.576097 5633 heartbeater.cc:507] Master 127.2.164.126:34679 requested a full tablet report, sending...
I20260202 15:03:40.577157 5168 ts_manager.cc:194] Registered new tserver with Master: 212239e7ef75440cb8d9fcaf9fc0ec55 (127.2.164.67:40805)
I20260202 15:03:40.577680 5168 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:59515
I20260202 15:03:40.582806 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260202 15:03:40.594439 5207 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:03:40.593083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.2.164.126:34679)
0202 15:03:40.593245 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:40.593252 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:40.593342 (+ 90us) server_negotiation.cc:408] Connection header received
0202 15:03:40.593510 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:40.593514 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:40.593567 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:40.593655 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:40.594334 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56794: BlockingRecv error: recv got EOF from 127.0.0.1:56794 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:03:40.595806 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5232
I20260202 15:03:40.601519 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5368
I20260202 15:03:40.607151 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5504
I20260202 15:03:40.612593 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5136
2026-02-02T15:03:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5950 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5645](info): set up 2 sockets
Feb 02 15:03:40 dist-test-slave-6zzp krb5kdc[5645](info): commencing operation
krb5kdc: starting...
W20260202 15:03:42.629428 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.994s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:42 dist-test-slave-6zzp krb5kdc[5645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:42.771958 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35141
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:42.887403 5661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:42.887694 5661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:42.887800 5661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:42.891207 5661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:42.891300 5661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:42.891345 5661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:42.891399 5661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:42.891443 5661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:42.895625 5661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:42.896749 5661 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:42.897629 5661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:42.903061 5666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:42.903048 5669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:42.903064 5667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:42.903316 5661 server_base.cc:1060] running on GCE node
I20260202 15:03:42.903872 5661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:42.904762 5661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:42.905966 5661 hybrid_clock.cc:648] HybridClock initialized: now 1770044622905959 us; error 31 us; skew 500 ppm
Feb 02 15:03:42 dist-test-slave-6zzp krb5kdc[5645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:42.908517 5661 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:42.909564 5661 webserver.cc:492] Webserver started at http://127.2.164.126:46753/ using document root <none> and password file <none>
I20260202 15:03:42.910142 5661 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:42.910213 5661 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:42.910414 5661 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:42.912060 5661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "cf690f38b7934cbeb49f228e4d31ccd3"
format_stamp: "Formatted at 2026-02-02 15:03:42 on dist-test-slave-6zzp"
server_key: "dc641c4286e4a893e99a64d9661cab7c"
server_key_iv: "a10d5f025ea4a6021ca0e1bf68cfdd55"
server_key_version: "encryptionkey@0"
I20260202 15:03:42.912544 5661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "cf690f38b7934cbeb49f228e4d31ccd3"
format_stamp: "Formatted at 2026-02-02 15:03:42 on dist-test-slave-6zzp"
server_key: "dc641c4286e4a893e99a64d9661cab7c"
server_key_iv: "a10d5f025ea4a6021ca0e1bf68cfdd55"
server_key_version: "encryptionkey@0"
I20260202 15:03:42.915807 5661 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260202 15:03:42.918334 5676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:42.919304 5661 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:42.919431 5661 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "cf690f38b7934cbeb49f228e4d31ccd3"
format_stamp: "Formatted at 2026-02-02 15:03:42 on dist-test-slave-6zzp"
server_key: "dc641c4286e4a893e99a64d9661cab7c"
server_key_iv: "a10d5f025ea4a6021ca0e1bf68cfdd55"
server_key_version: "encryptionkey@0"
I20260202 15:03:42.919529 5661 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:42.938452 5661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:42.941253 5661 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:42.941469 5661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:42.948300 5661 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35141
I20260202 15:03:42.948303 5728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:35141 every 8 connection(s)
I20260202 15:03:42.949522 5661 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:42.952414 5729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:42.957847 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3: Bootstrap starting.
I20260202 15:03:42.958774 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5661
I20260202 15:03:42.958884 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:42.959137 2705 external_mini_cluster.cc:1468] Setting key f64e3668acce82b9c3b04ef34c368156
I20260202 15:03:42.960390 5729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:42.961133 5729 log.cc:826] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:42.963097 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3: No bootstrap required, opened a new log
I20260202 15:03:42.965896 5729 raft_consensus.cc:359] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35141 } }
I20260202 15:03:42.966099 5729 raft_consensus.cc:385] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:42.966193 5729 raft_consensus.cc:740] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf690f38b7934cbeb49f228e4d31ccd3, State: Initialized, Role: FOLLOWER
Feb 02 15:03:42 dist-test-slave-6zzp krb5kdc[5645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:42.966706 5729 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35141 } }
I20260202 15:03:42.966869 5729 raft_consensus.cc:399] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:42.966957 5729 raft_consensus.cc:493] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:42.967048 5729 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:42.968118 5729 raft_consensus.cc:515] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35141 } }
I20260202 15:03:42.968461 5729 leader_election.cc:304] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf690f38b7934cbeb49f228e4d31ccd3; no voters:
I20260202 15:03:42.968732 5729 leader_election.cc:290] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:42.969317 5734 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:42.969599 5734 raft_consensus.cc:697] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [term 1 LEADER]: Becoming Leader. State: Replica: cf690f38b7934cbeb49f228e4d31ccd3, State: Running, Role: LEADER
I20260202 15:03:42.969973 5734 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35141 } }
I20260202 15:03:42.970206 5729 sys_catalog.cc:565] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:42.971529 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf690f38b7934cbeb49f228e4d31ccd3. Latest consensus state: current_term: 1 leader_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35141 } } }
I20260202 15:03:42.971676 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:42.971660 5735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf690f38b7934cbeb49f228e4d31ccd3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35141 } } }
I20260202 15:03:42.971846 5735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:42.972189 5744 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:42.972240 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:42.960161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.2.164.126:35141)
0202 15:03:42.960585 (+ 424us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:42.960594 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:42.960627 (+ 33us) server_negotiation.cc:408] Connection header received
0202 15:03:42.961245 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:42.961270 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:42.961563 (+ 293us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:42.961847 (+ 284us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:42.962660 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:42.963748 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:42.964411 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:42.964679 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:42.966824 (+ 2145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:42.966845 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:42.966896 (+ 51us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:42.966938 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:42.968800 (+ 1862us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:42.969450 (+ 650us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:42.969456 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:42.969461 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:42.969552 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:42.969838 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:42.969841 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:42.969843 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:42.970302 (+ 459us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:42.970485 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:42.970971 (+ 486us) server_negotiation.cc:300] Negotiation successful
0202 15:03:42.971266 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
I20260202 15:03:42.975886 5744 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:42.981210 5744 catalog_manager.cc:1357] Generated new cluster ID: 22f353c57cc3480886a46139c40f54bf
I20260202 15:03:42.981354 5744 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:43.009217 5744 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:43.010170 5744 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:43.032481 5744 catalog_manager.cc:6027] T 00000000000000000000000000000000 P cf690f38b7934cbeb49f228e4d31ccd3: Generated new TSK 0
I20260202 15:03:43.033362 5744 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:43.099794 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35141
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:43.231004 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:43.231215 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:43.231273 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:43.234982 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:43.235092 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:43.235194 5757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:43.239567 5757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:43.240650 5757 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:43.241529 5757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:43.248003 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:43.248032 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:43.248221 5757 server_base.cc:1060] running on GCE node
W20260202 15:03:43.248032 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:43.248618 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:43.249220 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:43.250382 5757 hybrid_clock.cc:648] HybridClock initialized: now 1770044623250368 us; error 32 us; skew 500 ppm
Feb 02 15:03:43 dist-test-slave-6zzp krb5kdc[5645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:43.252894 5757 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:43.253906 5757 webserver.cc:492] Webserver started at http://127.2.164.65:39575/ using document root <none> and password file <none>
I20260202 15:03:43.254410 5757 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:43.254458 5757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:43.254608 5757 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:43.256213 5757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "978d078e50c4403796eeaf207ab356d5"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "daf1c792b173904be41a9f34b87b09b2"
server_key_iv: "42b0ef16cebb3429ef8ee15cfe09e905"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.256641 5757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "978d078e50c4403796eeaf207ab356d5"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "daf1c792b173904be41a9f34b87b09b2"
server_key_iv: "42b0ef16cebb3429ef8ee15cfe09e905"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.259940 5757 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:43.262185 5772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.263868 5757 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260202 15:03:43.264014 5757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "978d078e50c4403796eeaf207ab356d5"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "daf1c792b173904be41a9f34b87b09b2"
server_key_iv: "42b0ef16cebb3429ef8ee15cfe09e905"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.264122 5757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:43.278795 5757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:43.281685 5757 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:43.281878 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:43.282431 5757 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:43.283300 5757 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:43.283365 5757 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.283435 5757 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:43.283468 5757 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.292445 5757 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:37037
I20260202 15:03:43.292485 5885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:37037 every 8 connection(s)
I20260202 15:03:43.293391 5757 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:43.296160 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5757
I20260202 15:03:43.296281 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:43.296537 2705 external_mini_cluster.cc:1468] Setting key f0dbedb89b59ba61ce30b51e92512398
Feb 02 15:03:43 dist-test-slave-6zzp krb5kdc[5645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:43.306386 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:43.295248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:60885 (local address 127.2.164.126:35141)
0202 15:03:43.295414 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.295419 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.296121 (+ 702us) server_negotiation.cc:408] Connection header received
0202 15:03:43.297129 (+ 1008us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.297132 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.297192 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.297294 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.298877 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.299387 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:43.300014 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.300194 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:43.302371 (+ 2177us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:43.302401 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:43.302408 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:43.302447 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:43.304333 (+ 1886us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:43.304843 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:43.304849 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:43.304852 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:43.304911 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:43.305259 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:43.305263 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:43.305265 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:43.305470 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:43.305592 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:43.306135 (+ 543us) server_negotiation.cc:300] Negotiation successful
0202 15:03:43.306255 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:03:43.307011 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:43.295552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35141 (local address 127.2.164.65:60885)
0202 15:03:43.295986 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:43.296020 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:43.296881 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:43.297449 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:43.297457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:43.297834 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:43.298738 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:43.298750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.299503 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:43.299511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:43.299901 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:43.299908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.300129 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:43.300723 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:43.300747 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:43.302197 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:43.304475 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:43.304480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:43.304496 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:43.304732 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:43.305064 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:43.305067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:43.305068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:43.305158 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:43.305602 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:43.305608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:43.305839 (+ 231us) client_negotiation.cc:769] Sending connection context
0202 15:03:43.306018 (+ 179us) client_negotiation.cc:240] Negotiation successful
0202 15:03:43.306256 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260202 15:03:43.308212 5886 heartbeater.cc:344] Connected to a master server at 127.2.164.126:35141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:03:43.308492 5886 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:43.309113 5886 heartbeater.cc:507] Master 127.2.164.126:35141 requested a full tablet report, sending...
I20260202 15:03:43.310642 5693 ts_manager.cc:194] Registered new tserver with Master: 978d078e50c4403796eeaf207ab356d5 (127.2.164.65:37037)
I20260202 15:03:43.312351 5693 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:60885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:43.351894 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35141
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:43.487205 5893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:43.487526 5893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:43.487637 5893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:43.493561 5893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:43.493667 5893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:43.493821 5893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:43.501108 5893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:43.502688 5893 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:43.503850 5893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:43.512228 5898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:43.512257 5899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:43.512228 5901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:43.512600 5893 server_base.cc:1060] running on GCE node
I20260202 15:03:43.513087 5893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:43.513687 5893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:43.514882 5893 hybrid_clock.cc:648] HybridClock initialized: now 1770044623514867 us; error 53 us; skew 500 ppm
Feb 02 15:03:43 dist-test-slave-6zzp krb5kdc[5645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:43.517730 5893 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:43.518783 5893 webserver.cc:492] Webserver started at http://127.2.164.66:40279/ using document root <none> and password file <none>
I20260202 15:03:43.519330 5893 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:43.519377 5893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:43.519536 5893 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:43.521301 5893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "ad8d6790db83466ebfcd4f97513ad2f9"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "4f7770ab047f55bbf0ec00116a2b5b00"
server_key_iv: "bd3321862177131a1f96eaed00b24148"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.521733 5893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "ad8d6790db83466ebfcd4f97513ad2f9"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "4f7770ab047f55bbf0ec00116a2b5b00"
server_key_iv: "bd3321862177131a1f96eaed00b24148"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.525261 5893 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:43.527575 5908 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.528666 5893 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:43.528805 5893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "ad8d6790db83466ebfcd4f97513ad2f9"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "4f7770ab047f55bbf0ec00116a2b5b00"
server_key_iv: "bd3321862177131a1f96eaed00b24148"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.528913 5893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:43.542963 5893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:43.546053 5893 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:43.546269 5893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:43.546885 5893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:43.547778 5893 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:43.547827 5893 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.547892 5893 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:43.547928 5893 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.557529 5893 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:38741
I20260202 15:03:43.557612 6021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:38741 every 8 connection(s)
I20260202 15:03:43.558543 5893 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:43.561457 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 5893
I20260202 15:03:43.561565 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:43.561815 2705 external_mini_cluster.cc:1468] Setting key 655d5a812e557f91dac62a3b4001712a
Feb 02 15:03:43 dist-test-slave-6zzp krb5kdc[5645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:03:43.572759 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:43.560447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:39075 (local address 127.2.164.126:35141)
0202 15:03:43.560624 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.560628 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.561409 (+ 781us) server_negotiation.cc:408] Connection header received
0202 15:03:43.562484 (+ 1075us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.562487 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.562560 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.562656 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.564881 (+ 2225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.565733 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:43.566444 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.566675 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:43.568782 (+ 2107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:43.568805 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:43.568810 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:43.568841 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:43.570624 (+ 1783us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:43.571161 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:43.571166 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:43.571168 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:43.571225 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:43.571614 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:43.571620 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:43.571624 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:43.571804 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:43.571927 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:43.572474 (+ 547us) server_negotiation.cc:300] Negotiation successful
0202 15:03:43.572626 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260202 15:03:43.573585 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:43.560701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35141 (local address 127.2.164.66:39075)
0202 15:03:43.561231 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:43.561290 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:43.562250 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:43.562804 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:43.562811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:43.563392 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:43.564700 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:43.564722 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.565905 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:43.565908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:43.566311 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:43.566318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.566497 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:43.567110 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:43.567138 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:43.568632 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:43.570767 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:43.570772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:43.570783 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:43.571051 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:43.571372 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:43.571377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:43.571382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:43.571513 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:43.571959 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:43.571965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:43.572240 (+ 275us) client_negotiation.cc:769] Sending connection context
0202 15:03:43.572444 (+ 204us) client_negotiation.cc:240] Negotiation successful
0202 15:03:43.572791 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:03:43.574801 6022 heartbeater.cc:344] Connected to a master server at 127.2.164.126:35141
I20260202 15:03:43.575094 6022 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:43.575752 6022 heartbeater.cc:507] Master 127.2.164.126:35141 requested a full tablet report, sending...
I20260202 15:03:43.576948 5693 ts_manager.cc:194] Registered new tserver with Master: ad8d6790db83466ebfcd4f97513ad2f9 (127.2.164.66:38741)
I20260202 15:03:43.577589 5693 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:39075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:43.616344 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35141
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:43.723819 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:43.724077 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:43.724184 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:43.727494 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:43.727591 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:43.727717 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:43.731998 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:43.733266 6029 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:43.734130 6029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:43.741328 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:43.741328 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:43.741328 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:43.741770 6029 server_base.cc:1060] running on GCE node
I20260202 15:03:43.742165 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:43.742741 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:43.743944 6029 hybrid_clock.cc:648] HybridClock initialized: now 1770044623743939 us; error 31 us; skew 500 ppm
Feb 02 15:03:43 dist-test-slave-6zzp krb5kdc[5645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:43.746716 6029 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:43.747834 6029 webserver.cc:492] Webserver started at http://127.2.164.67:40459/ using document root <none> and password file <none>
I20260202 15:03:43.748428 6029 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:43.748503 6029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:43.748724 6029 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:43.750586 6029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "a1eea749f9a944ceb961f34a84e4e9fe"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "d84e5acc27d95030198d9d36f23ee3cd"
server_key_iv: "b394a1643a8488425af57d240c032dd3"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.751103 6029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "a1eea749f9a944ceb961f34a84e4e9fe"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "d84e5acc27d95030198d9d36f23ee3cd"
server_key_iv: "b394a1643a8488425af57d240c032dd3"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.754808 6029 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260202 15:03:43.757288 6044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.758322 6029 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:03:43.758461 6029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "a1eea749f9a944ceb961f34a84e4e9fe"
format_stamp: "Formatted at 2026-02-02 15:03:43 on dist-test-slave-6zzp"
server_key: "d84e5acc27d95030198d9d36f23ee3cd"
server_key_iv: "b394a1643a8488425af57d240c032dd3"
server_key_version: "encryptionkey@0"
I20260202 15:03:43.758576 6029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:43.779021 6029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:43.782362 6029 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:43.782595 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:43.783201 6029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:43.784121 6029 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:43.784193 6029 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.784264 6029 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:43.784314 6029 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:43.793752 6029 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45457
I20260202 15:03:43.793766 6157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:45457 every 8 connection(s)
I20260202 15:03:43.794708 6029 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:43.802575 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6029
I20260202 15:03:43.802728 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:43.803030 2705 external_mini_cluster.cc:1468] Setting key f26470e60df37a1a33a7b71cd814c9e7
Feb 02 15:03:43 dist-test-slave-6zzp krb5kdc[5645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:43.807622 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:43.796487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:36465 (local address 127.2.164.126:35141)
0202 15:03:43.796640 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.796644 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.797401 (+ 757us) server_negotiation.cc:408] Connection header received
0202 15:03:43.798222 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.798226 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.798275 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.798351 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.800068 (+ 1717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.800584 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:43.801373 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.801503 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:43.804005 (+ 2502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:43.804028 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:43.804031 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:43.804079 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:43.805640 (+ 1561us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:43.806168 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:43.806177 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:43.806180 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:43.806224 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:43.806514 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:43.806518 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:43.806520 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:43.806685 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:43.806792 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:43.807319 (+ 527us) server_negotiation.cc:300] Negotiation successful
0202 15:03:43.807446 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260202 15:03:43.808276 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:43.796759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35141 (local address 127.2.164.67:36465)
0202 15:03:43.797255 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:43.797293 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:43.798013 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:43.798502 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:43.798510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:43.798905 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:43.799834 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:43.799849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.800741 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:43.800744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:43.801269 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:43.801278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:43.801469 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:43.802065 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:43.802091 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:43.803838 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:43.805772 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:43.805778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:43.805792 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:43.806052 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:43.806328 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:43.806331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:43.806332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:43.806422 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:43.806792 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:43.806801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:43.807066 (+ 265us) client_negotiation.cc:769] Sending connection context
0202 15:03:43.807288 (+ 222us) client_negotiation.cc:240] Negotiation successful
0202 15:03:43.807575 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
I20260202 15:03:43.809439 6158 heartbeater.cc:344] Connected to a master server at 127.2.164.126:35141
I20260202 15:03:43.809652 6158 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:43.810132 6158 heartbeater.cc:507] Master 127.2.164.126:35141 requested a full tablet report, sending...
I20260202 15:03:43.811184 5693 ts_manager.cc:194] Registered new tserver with Master: a1eea749f9a944ceb961f34a84e4e9fe (127.2.164.67:45457)
I20260202 15:03:43.811704 5693 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:36465
I20260202 15:03:43.816951 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260202 15:03:43.948289 5732 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:03:43.944758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.2.164.126:35141)
0202 15:03:43.944913 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.944917 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.945850 (+ 933us) server_negotiation.cc:408] Connection header received
0202 15:03:43.946775 (+ 925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.946779 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.946835 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.946952 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.948171 (+ 1219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52912: BlockingRecv error: recv got EOF from 127.0.0.1:52912 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260202 15:03:43.949811 5732 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:03:43.949100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.2.164.126:35141)
0202 15:03:43.949229 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.949234 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.949250 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:03:43.949294 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.949297 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.949341 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.949428 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.949729 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52914: BlockingRecv error: recv got EOF from 127.0.0.1:52914 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260202 15:03:43.950742 5732 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:03:43.950052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.2.164.126:35141)
0202 15:03:43.950139 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.950142 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.950181 (+ 39us) server_negotiation.cc:408] Connection header received
0202 15:03:43.950280 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.950283 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.950321 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.950396 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.950669 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52918: BlockingRecv error: recv got EOF from 127.0.0.1:52918 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260202 15:03:43.954391 5732 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:03:43.953652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.2.164.126:35141)
0202 15:03:43.953775 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:43.953780 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:43.953794 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:03:43.953881 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:43.953884 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:43.953920 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:43.953994 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:43.954314 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52920: BlockingRecv error: recv got EOF from 127.0.0.1:52920 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260202 15:03:43.962343 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5757
I20260202 15:03:43.967844 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5893
I20260202 15:03:43.973337 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6029
I20260202 15:03:43.978541 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 5661
2026-02-02T15:03:43Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3366 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:44 dist-test-slave-6zzp krb5kdc[6181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:44 dist-test-slave-6zzp krb5kdc[6181](info): set up 2 sockets
Feb 02 15:03:44 dist-test-slave-6zzp krb5kdc[6181](info): commencing operation
krb5kdc: starting...
W20260202 15:03:45.989401 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:46 dist-test-slave-6zzp krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:46.132462 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39557
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:39557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:46.237473 6197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:46.237749 6197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:46.237839 6197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:46.241236 6197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:46.241355 6197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:46.241405 6197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:46.241470 6197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:46.241514 6197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:46.245736 6197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:39557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:46.246840 6197 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:46.247679 6197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:46.253270 6203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:46.253356 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:46.253271 6205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:46.253549 6197 server_base.cc:1060] running on GCE node
I20260202 15:03:46.254040 6197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:46.254900 6197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:46.256098 6197 hybrid_clock.cc:648] HybridClock initialized: now 1770044626256080 us; error 42 us; skew 500 ppm
Feb 02 15:03:46 dist-test-slave-6zzp krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:46.258546 6197 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:46.259577 6197 webserver.cc:492] Webserver started at http://127.2.164.126:33655/ using document root <none> and password file <none>
I20260202 15:03:46.260130 6197 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:46.260203 6197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:46.260398 6197 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:46.262171 6197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "87cdb6fac2ee4d709f77700263256a21"
format_stamp: "Formatted at 2026-02-02 15:03:46 on dist-test-slave-6zzp"
server_key: "da254f81500a991949b5546cce4d056e"
server_key_iv: "b088ab4d02a6da258403fa9a4c15ef08"
server_key_version: "encryptionkey@0"
I20260202 15:03:46.262753 6197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "87cdb6fac2ee4d709f77700263256a21"
format_stamp: "Formatted at 2026-02-02 15:03:46 on dist-test-slave-6zzp"
server_key: "da254f81500a991949b5546cce4d056e"
server_key_iv: "b088ab4d02a6da258403fa9a4c15ef08"
server_key_version: "encryptionkey@0"
I20260202 15:03:46.266767 6197 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260202 15:03:46.269686 6212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:46.271204 6197 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260202 15:03:46.271376 6197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "87cdb6fac2ee4d709f77700263256a21"
format_stamp: "Formatted at 2026-02-02 15:03:46 on dist-test-slave-6zzp"
server_key: "da254f81500a991949b5546cce4d056e"
server_key_iv: "b088ab4d02a6da258403fa9a4c15ef08"
server_key_version: "encryptionkey@0"
I20260202 15:03:46.271492 6197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:46.297246 6197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:46.304237 6197 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:46.304461 6197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:46.312074 6197 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:39557
I20260202 15:03:46.312074 6264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:39557 every 8 connection(s)
I20260202 15:03:46.313374 6197 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:46.317273 6265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:46.318310 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6197
I20260202 15:03:46.318434 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:46.318697 2705 external_mini_cluster.cc:1468] Setting key f00f65ab7a20b333639f7e46e4672f44
I20260202 15:03:46.324653 6265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21: Bootstrap starting.
Feb 02 15:03:46 dist-test-slave-6zzp krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:46.327255 6265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:46.328122 6265 log.cc:826] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:46.330441 6265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21: No bootstrap required, opened a new log
I20260202 15:03:46.332520 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:46.320072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.2.164.126:39557)
0202 15:03:46.320577 (+ 505us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:46.320588 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:46.320627 (+ 39us) server_negotiation.cc:408] Connection header received
0202 15:03:46.321634 (+ 1007us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:46.321658 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:46.322112 (+ 454us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:46.322515 (+ 403us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:46.323477 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:46.324249 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:46.324986 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:46.325333 (+ 347us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:46.327867 (+ 2534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:46.327887 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:46.327898 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:46.327960 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:46.329964 (+ 2004us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:46.330440 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:46.330445 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:46.330450 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:46.330566 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:46.330816 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:46.330820 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:46.330822 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:46.331179 (+ 357us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:46.331395 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:46.331682 (+ 287us) server_negotiation.cc:300] Negotiation successful
0202 15:03:46.331929 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":157,"threads_started":1}
I20260202 15:03:46.333261 6265 raft_consensus.cc:359] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87cdb6fac2ee4d709f77700263256a21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39557 } }
I20260202 15:03:46.333503 6265 raft_consensus.cc:385] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:46.333621 6265 raft_consensus.cc:740] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cdb6fac2ee4d709f77700263256a21, State: Initialized, Role: FOLLOWER
I20260202 15:03:46.334101 6265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87cdb6fac2ee4d709f77700263256a21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39557 } }
I20260202 15:03:46.334263 6265 raft_consensus.cc:399] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:46.334383 6265 raft_consensus.cc:493] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:46.334504 6265 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:46.335402 2705 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260202 15:03:46.335636 6265 raft_consensus.cc:515] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87cdb6fac2ee4d709f77700263256a21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39557 } }
I20260202 15:03:46.336099 6265 leader_election.cc:304] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87cdb6fac2ee4d709f77700263256a21; no voters:
I20260202 15:03:46.336498 6265 leader_election.cc:290] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:46.336798 6270 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:46.337059 6270 raft_consensus.cc:697] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [term 1 LEADER]: Becoming Leader. State: Replica: 87cdb6fac2ee4d709f77700263256a21, State: Running, Role: LEADER
I20260202 15:03:46.337404 6270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87cdb6fac2ee4d709f77700263256a21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39557 } }
I20260202 15:03:46.337749 6265 sys_catalog.cc:565] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:46.339192 6271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87cdb6fac2ee4d709f77700263256a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87cdb6fac2ee4d709f77700263256a21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39557 } } }
I20260202 15:03:46.339215 6272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87cdb6fac2ee4d709f77700263256a21. Latest consensus state: current_term: 1 leader_uuid: "87cdb6fac2ee4d709f77700263256a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87cdb6fac2ee4d709f77700263256a21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39557 } } }
I20260202 15:03:46.339313 6271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:46.339314 6272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:46.339685 6280 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:46.342955 6280 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:46.348666 6280 catalog_manager.cc:1357] Generated new cluster ID: 3c1fade20f55479f96da01c9dd1395a3
I20260202 15:03:46.348778 6280 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:46.358682 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:46.353473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.2.164.126:39557)
0202 15:03:46.353588 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:46.353592 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:46.353854 (+ 262us) server_negotiation.cc:408] Connection header received
0202 15:03:46.353970 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:46.353981 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:46.354038 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:46.354130 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:46.354859 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:46.355661 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:46.356488 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:46.356693 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:46.357354 (+ 661us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:46.357373 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:46.357377 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:46.358205 (+ 828us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:46.358251 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:46.358270 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:46.358376 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:46.358478 (+ 102us) server_negotiation.cc:300] Negotiation successful
0202 15:03:46.358535 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260202 15:03:46.386314 6280 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:46.387861 6280 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:46.405592 6280 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 87cdb6fac2ee4d709f77700263256a21: Generated new TSK 0
I20260202 15:03:46.406520 6280 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:03:46.418849 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6197
2026-02-02T15:03:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2438 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:46 dist-test-slave-6zzp krb5kdc[6298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:46 dist-test-slave-6zzp krb5kdc[6298](info): set up 2 sockets
Feb 02 15:03:46 dist-test-slave-6zzp krb5kdc[6298](info): commencing operation
krb5kdc: starting...
W20260202 15:03:48.433531 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.994s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:48 dist-test-slave-6zzp krb5kdc[6298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:48.586434 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38647
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:48.689481 6314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:48.689733 6314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:48.689786 6314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:48.692981 6314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:48.693065 6314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:48.693094 6314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:48.693113 6314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:48.693130 6314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:48.697474 6314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:48.698515 6314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:48.699334 6314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:48.705156 6322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:48.705112 6319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:48.705186 6320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:48.705302 6314 server_base.cc:1060] running on GCE node
I20260202 15:03:48.705876 6314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:48.706733 6314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:48.707906 6314 hybrid_clock.cc:648] HybridClock initialized: now 1770044628707877 us; error 41 us; skew 500 ppm
Feb 02 15:03:48 dist-test-slave-6zzp krb5kdc[6298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:48.710840 6314 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:48.711939 6314 webserver.cc:492] Webserver started at http://127.2.164.126:46115/ using document root <none> and password file <none>
I20260202 15:03:48.712518 6314 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:48.712594 6314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:48.712805 6314 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:48.714584 6314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28"
format_stamp: "Formatted at 2026-02-02 15:03:48 on dist-test-slave-6zzp"
server_key: "305b199b0efd2c246e79d73bc66096fc"
server_key_iv: "f70464210e88dbbfc0ab12d75ee3c50d"
server_key_version: "encryptionkey@0"
I20260202 15:03:48.715263 6314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28"
format_stamp: "Formatted at 2026-02-02 15:03:48 on dist-test-slave-6zzp"
server_key: "305b199b0efd2c246e79d73bc66096fc"
server_key_iv: "f70464210e88dbbfc0ab12d75ee3c50d"
server_key_version: "encryptionkey@0"
I20260202 15:03:48.718919 6314 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:48.721284 6329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:48.722312 6314 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:48.722456 6314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28"
format_stamp: "Formatted at 2026-02-02 15:03:48 on dist-test-slave-6zzp"
server_key: "305b199b0efd2c246e79d73bc66096fc"
server_key_iv: "f70464210e88dbbfc0ab12d75ee3c50d"
server_key_version: "encryptionkey@0"
I20260202 15:03:48.722568 6314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:48.761786 6314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:48.764645 6314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:48.764855 6314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:48.772819 6314 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38647
I20260202 15:03:48.772866 6381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:38647 every 8 connection(s)
I20260202 15:03:48.773928 6314 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:48.777312 6382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:48.782341 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6314
I20260202 15:03:48.782508 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:48.782765 2705 external_mini_cluster.cc:1468] Setting key 1a7133b124d7060e4453fd11ec4abcd6
I20260202 15:03:48.783370 6382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28: Bootstrap starting.
I20260202 15:03:48.786194 6382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:48.787014 6382 log.cc:826] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:48.789160 6382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28: No bootstrap required, opened a new log
Feb 02 15:03:48 dist-test-slave-6zzp krb5kdc[6298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:48.792287 6382 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38647 } }
I20260202 15:03:48.792560 6382 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:48.792620 6382 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1b6cd1074d4ec5a03fcbbcf5244b28, State: Initialized, Role: FOLLOWER
I20260202 15:03:48.793190 6382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38647 } }
I20260202 15:03:48.793367 6382 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:48.793424 6382 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:48.793521 6382 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:48.794620 6382 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38647 } }
I20260202 15:03:48.795007 6382 leader_election.cc:304] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1b6cd1074d4ec5a03fcbbcf5244b28; no voters:
I20260202 15:03:48.795354 6382 leader_election.cc:290] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:48.795807 6387 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:48.796073 6387 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [term 1 LEADER]: Becoming Leader. State: Replica: 7d1b6cd1074d4ec5a03fcbbcf5244b28, State: Running, Role: LEADER
I20260202 15:03:48.796411 6387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38647 } }
I20260202 15:03:48.796649 6382 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:48.797890 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:48.784150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.2.164.126:38647)
0202 15:03:48.784621 (+ 471us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:48.784636 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:48.784683 (+ 47us) server_negotiation.cc:408] Connection header received
0202 15:03:48.785413 (+ 730us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:48.785439 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:48.785808 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:48.786194 (+ 386us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:48.787117 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:48.788105 (+ 988us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:48.788772 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:48.789076 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:48.791663 (+ 2587us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:48.791685 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:48.791699 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:48.791744 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:48.793968 (+ 2224us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:48.794484 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:48.794493 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:48.794502 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:48.794603 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:48.795140 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:48.795144 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:48.795145 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:48.795644 (+ 499us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:48.796575 (+ 931us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:48.796753 (+ 178us) server_negotiation.cc:300] Negotiation successful
0202 15:03:48.797098 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260202 15:03:48.798040 6388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38647 } } }
I20260202 15:03:48.798249 6388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:48.798067 6389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1b6cd1074d4ec5a03fcbbcf5244b28. Latest consensus state: current_term: 1 leader_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1b6cd1074d4ec5a03fcbbcf5244b28" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38647 } } }
I20260202 15:03:48.798525 6389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:48.798811 6397 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:48.802277 6397 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:48.807684 6397 catalog_manager.cc:1357] Generated new cluster ID: d7a7788dd0ba46798c7a39f7f43473e2
I20260202 15:03:48.807760 6397 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:48.837476 6397 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:48.838577 6397 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:48.845826 6397 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7d1b6cd1074d4ec5a03fcbbcf5244b28: Generated new TSK 0
I20260202 15:03:48.846498 6397 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:48.904840 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38647
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:49.014529 6410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:49.014752 6410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:49.014811 6410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:49.018369 6410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:49.018442 6410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:49.018570 6410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:49.022857 6410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:49.023936 6410 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:49.024784 6410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:49.031498 6418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:49.031513 6416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:49.031517 6415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:49.031908 6410 server_base.cc:1060] running on GCE node
I20260202 15:03:49.032327 6410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:49.032867 6410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:49.034085 6410 hybrid_clock.cc:648] HybridClock initialized: now 1770044629034081 us; error 54 us; skew 500 ppm
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:49.036880 6410 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:49.038031 6410 webserver.cc:492] Webserver started at http://127.2.164.65:46195/ using document root <none> and password file <none>
I20260202 15:03:49.038622 6410 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:49.038699 6410 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:49.038914 6410 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:49.040663 6410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "8d1afa06083b4458a10da6ae478e8ad3"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "b70a342ba7ce41ffc09874dcf3ca6730"
server_key_iv: "c6d239134a355aafcacb91b006310a8d"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.041208 6410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "8d1afa06083b4458a10da6ae478e8ad3"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "b70a342ba7ce41ffc09874dcf3ca6730"
server_key_iv: "c6d239134a355aafcacb91b006310a8d"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.044618 6410 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260202 15:03:49.046880 6425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.047883 6410 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:49.048043 6410 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "8d1afa06083b4458a10da6ae478e8ad3"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "b70a342ba7ce41ffc09874dcf3ca6730"
server_key_iv: "c6d239134a355aafcacb91b006310a8d"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.048156 6410 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:49.067857 6410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:49.070996 6410 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:49.071218 6410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:49.071786 6410 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:49.072700 6410 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:49.072775 6410 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.072846 6410 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:49.072898 6410 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.082338 6410 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:36963
I20260202 15:03:49.082357 6538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:36963 every 8 connection(s)
I20260202 15:03:49.083344 6410 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:49.091063 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6410
I20260202 15:03:49.091162 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:49.091439 2705 external_mini_cluster.cc:1468] Setting key 9d201e018de46bd5eab25ef6d9e04d1a
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:49.097090 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.085298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41395 (local address 127.2.164.126:38647)
0202 15:03:49.085458 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:49.085462 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:49.086198 (+ 736us) server_negotiation.cc:408] Connection header received
0202 15:03:49.087097 (+ 899us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:49.087101 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:49.087165 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:49.087266 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:49.088949 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.089534 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.090200 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.090389 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.093077 (+ 2688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:49.093094 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:49.093096 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:49.093122 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:49.094788 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.095369 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.095376 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.095380 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.095438 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.095773 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.095780 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.095784 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.096017 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:49.096124 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:49.096684 (+ 560us) server_negotiation.cc:300] Negotiation successful
0202 15:03:49.096841 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260202 15:03:49.097699 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.085586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38647 (local address 127.2.164.65:41395)
0202 15:03:49.086039 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:49.086077 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:49.086874 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:49.087440 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:49.087449 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:49.087836 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:49.088789 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:49.088803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.089660 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.089664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:49.090070 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:49.090077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.090326 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.090986 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:49.091013 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:49.092852 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:49.094931 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:49.094937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:49.094949 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:49.095253 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:49.095542 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:49.095545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:49.095547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:49.095669 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:49.096135 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:49.096141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:49.096380 (+ 239us) client_negotiation.cc:769] Sending connection context
0202 15:03:49.096613 (+ 233us) client_negotiation.cc:240] Negotiation successful
0202 15:03:49.096853 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260202 15:03:49.098964 6539 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38647
I20260202 15:03:49.099273 6539 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:49.099931 6539 heartbeater.cc:507] Master 127.2.164.126:38647 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:49.101641 6346 ts_manager.cc:194] Registered new tserver with Master: 8d1afa06083b4458a10da6ae478e8ad3 (127.2.164.65:36963)
I20260202 15:03:49.102797 6346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:41395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:49.142468 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38647
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:49.250689 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:49.250905 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:49.251004 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:49.254343 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:49.254405 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:49.254534 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:49.259027 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:49.260169 6546 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:49.261066 6546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:49.268019 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:49.268035 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:49.268019 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:49.268311 6546 server_base.cc:1060] running on GCE node
I20260202 15:03:49.268780 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:49.269400 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:49.270589 6546 hybrid_clock.cc:648] HybridClock initialized: now 1770044629270565 us; error 44 us; skew 500 ppm
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:49.273351 6546 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:49.274495 6546 webserver.cc:492] Webserver started at http://127.2.164.66:34543/ using document root <none> and password file <none>
I20260202 15:03:49.275084 6546 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:49.275159 6546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:49.275384 6546 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:49.277181 6546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "696ac385ccf841c4bb51c8a5b6a5baa6"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "4c35df444c264cdb3fdff96bda7ed088"
server_key_iv: "7761cdb6b75e4da06a8b90ef0e00548e"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.277663 6546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "696ac385ccf841c4bb51c8a5b6a5baa6"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "4c35df444c264cdb3fdff96bda7ed088"
server_key_iv: "7761cdb6b75e4da06a8b90ef0e00548e"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.280985 6546 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:49.283218 6561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.284221 6546 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260202 15:03:49.284356 6546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "696ac385ccf841c4bb51c8a5b6a5baa6"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "4c35df444c264cdb3fdff96bda7ed088"
server_key_iv: "7761cdb6b75e4da06a8b90ef0e00548e"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.284458 6546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:49.295169 6546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:49.298202 6546 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:49.298441 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:49.299021 6546 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:49.299935 6546 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:49.299983 6546 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.300020 6546 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:49.300034 6546 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.309800 6546 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:33561
I20260202 15:03:49.309837 6674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:33561 every 8 connection(s)
I20260202 15:03:49.310772 6546 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:49.318084 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6546
I20260202 15:03:49.318361 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:49.318624 2705 external_mini_cluster.cc:1468] Setting key 661ff56e660c66f115f5d341f054faa2
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:49.323855 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.312551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46043 (local address 127.2.164.126:38647)
0202 15:03:49.312692 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:49.312696 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:49.313457 (+ 761us) server_negotiation.cc:408] Connection header received
0202 15:03:49.314305 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:49.314310 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:49.314371 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:49.314472 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:49.316085 (+ 1613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.316593 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.317317 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.317497 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.320021 (+ 2524us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:49.320039 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:49.320042 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:49.320072 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:49.321678 (+ 1606us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.322222 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.322226 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.322228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.322287 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.322674 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.322677 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.322679 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.322840 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:49.322926 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:49.323593 (+ 667us) server_negotiation.cc:300] Negotiation successful
0202 15:03:49.323697 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:03:49.324774 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.312842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38647 (local address 127.2.164.66:46043)
0202 15:03:49.313309 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:49.313345 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:49.314095 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:49.314626 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:49.314634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:49.315010 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:49.315936 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:49.315950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.316716 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.316722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:49.317201 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:49.317209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.317443 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.318144 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:49.318168 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:49.319864 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:49.321813 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:49.321820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:49.321831 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:49.322093 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:49.322391 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:49.322398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:49.322403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:49.322567 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:49.322960 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:49.322971 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:49.323304 (+ 333us) client_negotiation.cc:769] Sending connection context
0202 15:03:49.323559 (+ 255us) client_negotiation.cc:240] Negotiation successful
0202 15:03:49.323864 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260202 15:03:49.326143 6675 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38647
I20260202 15:03:49.326412 6675 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:49.326987 6675 heartbeater.cc:507] Master 127.2.164.126:38647 requested a full tablet report, sending...
I20260202 15:03:49.328125 6346 ts_manager.cc:194] Registered new tserver with Master: 696ac385ccf841c4bb51c8a5b6a5baa6 (127.2.164.66:33561)
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:03:49.328909 6346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:46043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:49.368345 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38647
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:49.478931 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:49.479236 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:49.479332 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:49.482623 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:49.482722 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:49.482837 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:49.486896 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:49.487957 6682 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:49.488843 6682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:49.495491 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:49.495491 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:49.495688 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:49.495679 6682 server_base.cc:1060] running on GCE node
I20260202 15:03:49.496233 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:49.496794 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:49.497988 6682 hybrid_clock.cc:648] HybridClock initialized: now 1770044629497959 us; error 50 us; skew 500 ppm
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:49.501077 6682 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:49.502045 6682 webserver.cc:492] Webserver started at http://127.2.164.67:41687/ using document root <none> and password file <none>
I20260202 15:03:49.502560 6682 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:49.502607 6682 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:49.502763 6682 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:49.504479 6682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "dd6ee02564534b7bbefd2315a4c976f8"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "17f7028d5eb1e71edf704abaa19e8d4a"
server_key_iv: "82ab776fba92e7d7281051b9af1248d0"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.505149 6682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "dd6ee02564534b7bbefd2315a4c976f8"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "17f7028d5eb1e71edf704abaa19e8d4a"
server_key_iv: "82ab776fba92e7d7281051b9af1248d0"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.508725 6682 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:49.511123 6697 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.512282 6682 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:49.512426 6682 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "dd6ee02564534b7bbefd2315a4c976f8"
format_stamp: "Formatted at 2026-02-02 15:03:49 on dist-test-slave-6zzp"
server_key: "17f7028d5eb1e71edf704abaa19e8d4a"
server_key_iv: "82ab776fba92e7d7281051b9af1248d0"
server_key_version: "encryptionkey@0"
I20260202 15:03:49.512531 6682 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:49.531973 6682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:49.535171 6682 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:49.535388 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:49.535974 6682 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:49.536893 6682 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:49.536942 6682 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.537043 6682 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:49.537102 6682 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:49.547103 6682 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:42961
I20260202 15:03:49.547125 6810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:42961 every 8 connection(s)
I20260202 15:03:49.548069 6682 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:49.554391 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6682
I20260202 15:03:49.554517 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:49.554765 2705 external_mini_cluster.cc:1468] Setting key 3ddd28a7749bcd34f55a60908bb4a760
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:49.560637 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.550122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:58127 (local address 127.2.164.126:38647)
0202 15:03:49.550264 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:49.550268 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:49.550929 (+ 661us) server_negotiation.cc:408] Connection header received
0202 15:03:49.551742 (+ 813us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:49.551746 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:49.551797 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:49.551910 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:49.553586 (+ 1676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.554107 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.554993 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.555177 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.557660 (+ 2483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:49.557674 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:49.557676 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:49.557699 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:49.559048 (+ 1349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.559492 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.559494 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.559496 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.559535 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.559779 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.559781 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.559782 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.559913 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:49.559983 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:49.560430 (+ 447us) server_negotiation.cc:300] Negotiation successful
0202 15:03:49.560521 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:03:49.561348 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.550386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38647 (local address 127.2.164.67:58127)
0202 15:03:49.550786 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:49.550822 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:49.551517 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:49.552028 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:49.552035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:49.552427 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:49.553447 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:49.553460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.554275 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.554281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:49.554871 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:49.554881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.555116 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.555757 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:49.555789 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:49.557533 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:49.559165 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:49.559169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:49.559177 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:49.559402 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:49.559609 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:49.559612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:49.559613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:49.559706 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:49.559994 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:49.559999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:49.560222 (+ 223us) client_negotiation.cc:769] Sending connection context
0202 15:03:49.560390 (+ 168us) client_negotiation.cc:240] Negotiation successful
0202 15:03:49.560589 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
I20260202 15:03:49.562382 6811 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38647
I20260202 15:03:49.562625 6811 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:49.563123 6811 heartbeater.cc:507] Master 127.2.164.126:38647 requested a full tablet report, sending...
I20260202 15:03:49.564055 6346 ts_manager.cc:194] Registered new tserver with Master: dd6ee02564534b7bbefd2315a4c976f8 (127.2.164.67:42961)
I20260202 15:03:49.564608 6346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:58127
I20260202 15:03:49.568298 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044629, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:49.592751 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:49.584613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.2.164.126:38647)
0202 15:03:49.584752 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:49.584756 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:49.584866 (+ 110us) server_negotiation.cc:408] Connection header received
0202 15:03:49.584989 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:49.584992 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:49.585109 (+ 117us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:49.585184 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:49.586043 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.586544 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:49.587236 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:49.587432 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:49.589809 (+ 2377us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:49.589829 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:49.589831 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:49.589858 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:49.591305 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.591680 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.591683 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.591684 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.591728 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:49.591963 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:49.591966 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:49.591968 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:49.592185 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:49.592304 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:49.592521 (+ 217us) server_negotiation.cc:300] Negotiation successful
0202 15:03:49.592626 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260202 15:03:49.593250 6346 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50420
I20260202 15:03:49.594120 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6410
I20260202 15:03:49.600284 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6546
I20260202 15:03:49.605891 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6682
I20260202 15:03:49.610987 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6314
2026-02-02T15:03:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3193 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6823](info): set up 2 sockets
Feb 02 15:03:49 dist-test-slave-6zzp krb5kdc[6823](info): commencing operation
krb5kdc: starting...
W20260202 15:03:51.629663 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.997s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:51 dist-test-slave-6zzp krb5kdc[6823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:51.770350 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33269
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:51.878705 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:51.878935 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:51.878988 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:51.882337 6839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:51.882402 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:51.882432 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:51.882498 6839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:51.882536 6839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:51.886718 6839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:51.887785 6839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:51.888641 6839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:51.894663 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:51.894724 6839 server_base.cc:1060] running on GCE node
W20260202 15:03:51.894660 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:51.894654 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:51.895419 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:51.896446 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:51.897626 6839 hybrid_clock.cc:648] HybridClock initialized: now 1770044631897569 us; error 98 us; skew 500 ppm
Feb 02 15:03:51 dist-test-slave-6zzp krb5kdc[6823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:51.900198 6839 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:51.901293 6839 webserver.cc:492] Webserver started at http://127.2.164.126:38603/ using document root <none> and password file <none>
I20260202 15:03:51.901816 6839 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:51.901865 6839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:51.902020 6839 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:51.903659 6839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "52c85f96640840a8b210a1aa68eff87a"
format_stamp: "Formatted at 2026-02-02 15:03:51 on dist-test-slave-6zzp"
server_key: "91eb544340ba48c9a6d1f5b493086005"
server_key_iv: "050e5a76d7f086d855bdb6e33862f0fb"
server_key_version: "encryptionkey@0"
I20260202 15:03:51.904091 6839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "52c85f96640840a8b210a1aa68eff87a"
format_stamp: "Formatted at 2026-02-02 15:03:51 on dist-test-slave-6zzp"
server_key: "91eb544340ba48c9a6d1f5b493086005"
server_key_iv: "050e5a76d7f086d855bdb6e33862f0fb"
server_key_version: "encryptionkey@0"
I20260202 15:03:51.907526 6839 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260202 15:03:51.909761 6854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:51.910876 6839 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:51.910970 6839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "52c85f96640840a8b210a1aa68eff87a"
format_stamp: "Formatted at 2026-02-02 15:03:51 on dist-test-slave-6zzp"
server_key: "91eb544340ba48c9a6d1f5b493086005"
server_key_iv: "050e5a76d7f086d855bdb6e33862f0fb"
server_key_version: "encryptionkey@0"
I20260202 15:03:51.911051 6839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:51.926743 6839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:51.929718 6839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:51.929868 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:51.937088 6839 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:33269
I20260202 15:03:51.937150 6906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:33269 every 8 connection(s)
I20260202 15:03:51.938081 6839 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:51.940757 6907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:51.946174 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a: Bootstrap starting.
I20260202 15:03:51.946539 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6839
I20260202 15:03:51.946679 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:51.946923 2705 external_mini_cluster.cc:1468] Setting key bbc17e696a9062e38cfbdf9eb9224a2f
I20260202 15:03:51.948711 6907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:51.949574 6907 log.cc:826] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:51.951694 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a: No bootstrap required, opened a new log
I20260202 15:03:51.954815 6907 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c85f96640840a8b210a1aa68eff87a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33269 } }
I20260202 15:03:51.955044 6907 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:51.955108 6907 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c85f96640840a8b210a1aa68eff87a, State: Initialized, Role: FOLLOWER
Feb 02 15:03:51 dist-test-slave-6zzp krb5kdc[6823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:51.955569 6907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c85f96640840a8b210a1aa68eff87a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33269 } }
I20260202 15:03:51.955686 6907 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:51.955765 6907 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:51.955852 6907 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:51.957062 6907 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c85f96640840a8b210a1aa68eff87a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33269 } }
I20260202 15:03:51.957477 6907 leader_election.cc:304] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c85f96640840a8b210a1aa68eff87a; no voters:
I20260202 15:03:51.957830 6907 leader_election.cc:290] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:51.958320 6912 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:51.958575 6912 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [term 1 LEADER]: Becoming Leader. State: Replica: 52c85f96640840a8b210a1aa68eff87a, State: Running, Role: LEADER
I20260202 15:03:51.958882 6912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c85f96640840a8b210a1aa68eff87a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33269 } }
I20260202 15:03:51.959072 6907 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:51.960457 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c85f96640840a8b210a1aa68eff87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c85f96640840a8b210a1aa68eff87a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33269 } } }
I20260202 15:03:51.960582 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:51.961104 6914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c85f96640840a8b210a1aa68eff87a. Latest consensus state: current_term: 1 leader_uuid: "52c85f96640840a8b210a1aa68eff87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c85f96640840a8b210a1aa68eff87a" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33269 } } }
I20260202 15:03:51.961205 6914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:51.961547 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:51.948154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.2.164.126:33269)
0202 15:03:51.948621 (+ 467us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:51.948634 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:51.948677 (+ 43us) server_negotiation.cc:408] Connection header received
0202 15:03:51.949380 (+ 703us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:51.949402 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:51.949756 (+ 354us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:51.950556 (+ 800us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:51.951453 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:51.952248 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:51.953328 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:51.953654 (+ 326us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:51.955982 (+ 2328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:51.956006 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:51.956022 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:51.956073 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:51.958228 (+ 2155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:51.958732 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:51.958737 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:51.958742 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:51.958829 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:51.959143 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:51.959148 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:51.959150 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:51.959578 (+ 428us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:51.959806 (+ 228us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:51.960440 (+ 634us) server_negotiation.cc:300] Negotiation successful
0202 15:03:51.960700 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
W20260202 15:03:51.964241 6927 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:03:51.964334 6927 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:03:51.964396 6922 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:51.965335 6922 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:51.970489 6922 catalog_manager.cc:1357] Generated new cluster ID: 3493c2c61f3a43d48437a2502ecac710
I20260202 15:03:51.970561 6922 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:51.980319 6922 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:51.981578 6922 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:51.999498 6922 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 52c85f96640840a8b210a1aa68eff87a: Generated new TSK 0
I20260202 15:03:52.000413 6922 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:52.049975 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33269
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:52.156401 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:52.156653 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:52.156744 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:52.160199 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:52.160283 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:52.160401 6935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:52.164624 6935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:52.165774 6935 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:52.166620 6935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:52.173458 6940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:52.173641 6935 server_base.cc:1060] running on GCE node
W20260202 15:03:52.173458 6943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:52.173460 6941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:52.174273 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:52.174831 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:52.176002 6935 hybrid_clock.cc:648] HybridClock initialized: now 1770044632175974 us; error 49 us; skew 500 ppm
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:52.178603 6935 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:52.179656 6935 webserver.cc:492] Webserver started at http://127.2.164.65:35895/ using document root <none> and password file <none>
I20260202 15:03:52.180208 6935 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:52.180282 6935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:52.180498 6935 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:52.182257 6935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "9e1d4c24662a4985a635b949df65a366"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "9e3583699e9624bcbfa9cfde238db988"
server_key_iv: "c83516145217afe0015ab43b82c6c5e2"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.182778 6935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "9e1d4c24662a4985a635b949df65a366"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "9e3583699e9624bcbfa9cfde238db988"
server_key_iv: "c83516145217afe0015ab43b82c6c5e2"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.186250 6935 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260202 15:03:52.188406 6950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.189639 6935 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:52.189775 6935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "9e1d4c24662a4985a635b949df65a366"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "9e3583699e9624bcbfa9cfde238db988"
server_key_iv: "c83516145217afe0015ab43b82c6c5e2"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.189885 6935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:52.208734 6935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:52.211834 6935 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:52.212044 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:52.212590 6935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:52.213485 6935 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:52.213555 6935 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.213629 6935 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:52.213680 6935 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.222836 6935 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:46475
I20260202 15:03:52.222874 7063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:46475 every 8 connection(s)
I20260202 15:03:52.223773 6935 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:52.225867 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 6935
I20260202 15:03:52.225988 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:52.226274 2705 external_mini_cluster.cc:1468] Setting key b41fa943b4bc0e969583e5f409a793a2
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:52.237658 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.225524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:54413 (local address 127.2.164.126:33269)
0202 15:03:52.225696 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:52.225704 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:52.226428 (+ 724us) server_negotiation.cc:408] Connection header received
0202 15:03:52.227269 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:52.227273 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:52.227316 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:52.227406 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:52.229698 (+ 2292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.230524 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.231200 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.231440 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.234061 (+ 2621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:52.234088 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:52.234093 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:52.234130 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:52.235774 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.236301 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.236305 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.236306 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.236349 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.236597 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.236599 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.236600 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.236742 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:52.236817 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:52.237422 (+ 605us) server_negotiation.cc:300] Negotiation successful
0202 15:03:52.237536 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:03:52.238404 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.225739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33269 (local address 127.2.164.65:54413)
0202 15:03:52.226264 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:52.226311 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:52.227086 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:52.227609 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:52.227617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:52.228163 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:52.229498 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:52.229522 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.230666 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.230669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:52.231070 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:52.231077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.231259 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.231948 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:52.231972 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:52.233903 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:52.235930 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:52.235937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:52.235950 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:52.236192 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:52.236437 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:52.236440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:52.236441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:52.236520 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:52.236827 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:52.236832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:52.237202 (+ 370us) client_negotiation.cc:769] Sending connection context
0202 15:03:52.237379 (+ 177us) client_negotiation.cc:240] Negotiation successful
0202 15:03:52.237591 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:03:52.239650 7064 heartbeater.cc:344] Connected to a master server at 127.2.164.126:33269
I20260202 15:03:52.239917 7064 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:52.240501 7064 heartbeater.cc:507] Master 127.2.164.126:33269 requested a full tablet report, sending...
I20260202 15:03:52.242024 6870 ts_manager.cc:194] Registered new tserver with Master: 9e1d4c24662a4985a635b949df65a366 (127.2.164.65:46475)
I20260202 15:03:52.243203 6870 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:54413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:52.277366 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33269
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:52.384395 7071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:52.384624 7071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:52.384732 7071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:52.388002 7071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:52.388063 7071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:52.388195 7071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:52.392345 7071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:52.393409 7071 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:52.394366 7071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:52.401072 7076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:52.401072 7079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:52.401225 7071 server_base.cc:1060] running on GCE node
W20260202 15:03:52.401098 7077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:52.401759 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:52.402320 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:52.403498 7071 hybrid_clock.cc:648] HybridClock initialized: now 1770044632403473 us; error 40 us; skew 500 ppm
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:52.406088 7071 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:52.407115 7071 webserver.cc:492] Webserver started at http://127.2.164.66:44623/ using document root <none> and password file <none>
I20260202 15:03:52.407639 7071 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:52.407687 7071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:52.407837 7071 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:52.409580 7071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "8b2b833fe6d64767b2c51e565163a3d3"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "963920d00892e7006b672a7d5642fad1"
server_key_iv: "9f70be5b73613f52676b982bd243d064"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.410004 7071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "8b2b833fe6d64767b2c51e565163a3d3"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "963920d00892e7006b672a7d5642fad1"
server_key_iv: "9f70be5b73613f52676b982bd243d064"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.413604 7071 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:03:52.415953 7086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.417104 7071 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:52.417244 7071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "8b2b833fe6d64767b2c51e565163a3d3"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "963920d00892e7006b672a7d5642fad1"
server_key_iv: "9f70be5b73613f52676b982bd243d064"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.417357 7071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:52.456113 7071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:52.459365 7071 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:52.459596 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:52.460302 7071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:52.461522 7071 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:52.461571 7071 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.461638 7071 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:52.461669 7071 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.470952 7071 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:43339
I20260202 15:03:52.470973 7199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:43339 every 8 connection(s)
I20260202 15:03:52.471921 7071 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:52.473630 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7071
I20260202 15:03:52.473763 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:52.474025 2705 external_mini_cluster.cc:1468] Setting key bc130afa22b8cd2a414d00577c68d0fb
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:03:52.485306 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.473781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:34293 (local address 127.2.164.126:33269)
0202 15:03:52.473932 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:52.473937 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:52.474731 (+ 794us) server_negotiation.cc:408] Connection header received
0202 15:03:52.475712 (+ 981us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:52.475715 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:52.475756 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:52.475825 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:52.477922 (+ 2097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.478714 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.479446 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.479676 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.481924 (+ 2248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:52.481946 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:52.481952 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:52.481984 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:52.483585 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.484023 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.484026 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.484027 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.484068 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.484302 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.484304 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.484306 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.484500 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:52.484576 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:52.485083 (+ 507us) server_negotiation.cc:300] Negotiation successful
0202 15:03:52.485181 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:03:52.486151 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.474049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33269 (local address 127.2.164.66:34293)
0202 15:03:52.474576 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:52.474624 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:52.475509 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:52.476011 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:52.476023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:52.476611 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:52.477775 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:52.477790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.478834 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.478837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:52.479318 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:52.479326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.479494 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.480075 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:52.480099 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:52.481765 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:52.483700 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:52.483704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:52.483713 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:52.483933 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:52.484145 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:52.484147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:52.484149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:52.484228 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:52.484594 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:52.484600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:52.484824 (+ 224us) client_negotiation.cc:769] Sending connection context
0202 15:03:52.485043 (+ 219us) client_negotiation.cc:240] Negotiation successful
0202 15:03:52.485324 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":121,"threads_started":1}
I20260202 15:03:52.487287 7200 heartbeater.cc:344] Connected to a master server at 127.2.164.126:33269
I20260202 15:03:52.487562 7200 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:52.488091 7200 heartbeater.cc:507] Master 127.2.164.126:33269 requested a full tablet report, sending...
I20260202 15:03:52.489223 6870 ts_manager.cc:194] Registered new tserver with Master: 8b2b833fe6d64767b2c51e565163a3d3 (127.2.164.66:43339)
I20260202 15:03:52.489782 6870 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:34293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:52.528723 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33269
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:52.646147 7207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:52.646378 7207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:52.646437 7207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:52.649715 7207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:52.649780 7207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:52.649858 7207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:52.654127 7207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:52.655449 7207 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:52.656335 7207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:52.663939 7215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:52.663939 7213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:52.663939 7212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:52.664476 7207 server_base.cc:1060] running on GCE node
I20260202 15:03:52.664892 7207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:52.665570 7207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:52.666797 7207 hybrid_clock.cc:648] HybridClock initialized: now 1770044632666771 us; error 41 us; skew 500 ppm
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:52.669762 7207 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:52.670996 7207 webserver.cc:492] Webserver started at http://127.2.164.67:44653/ using document root <none> and password file <none>
I20260202 15:03:52.671617 7207 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:52.671694 7207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:52.671907 7207 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:52.673709 7207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "52c52033d6044764af816b89e21da2a5"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "8f1fd873b588c1b1c8efc8cbcdae7b40"
server_key_iv: "099202f3ddadfcd5de17f6149c583200"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.674207 7207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "52c52033d6044764af816b89e21da2a5"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "8f1fd873b588c1b1c8efc8cbcdae7b40"
server_key_iv: "099202f3ddadfcd5de17f6149c583200"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.677776 7207 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:52.679996 7222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.681134 7207 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:52.681257 7207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "52c52033d6044764af816b89e21da2a5"
format_stamp: "Formatted at 2026-02-02 15:03:52 on dist-test-slave-6zzp"
server_key: "8f1fd873b588c1b1c8efc8cbcdae7b40"
server_key_iv: "099202f3ddadfcd5de17f6149c583200"
server_key_version: "encryptionkey@0"
I20260202 15:03:52.681367 7207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:52.699326 7207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:52.702697 7207 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:52.702900 7207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:52.703467 7207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:52.704346 7207 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:52.704427 7207 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.704507 7207 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:52.704557 7207 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:52.713522 7207 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:33693
I20260202 15:03:52.713552 7335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:33693 every 8 connection(s)
I20260202 15:03:52.714465 7207 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:52.715173 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7207
I20260202 15:03:52.715302 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:52.715567 2705 external_mini_cluster.cc:1468] Setting key a535f2599fa2eb9be2c5e2e1e784516a
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:52.727329 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.717142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:57105 (local address 127.2.164.126:33269)
0202 15:03:52.717274 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:52.717279 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:52.717525 (+ 246us) server_negotiation.cc:408] Connection header received
0202 15:03:52.718435 (+ 910us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:52.718438 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:52.718496 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:52.718571 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:52.720170 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.720677 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.721356 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.721563 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.723798 (+ 2235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:52.723816 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:52.723819 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:52.723845 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:52.725341 (+ 1496us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.725871 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.725874 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.725876 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.725923 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.726258 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.726261 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.726262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.726439 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:52.726523 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:52.727081 (+ 558us) server_negotiation.cc:300] Negotiation successful
0202 15:03:52.727196 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260202 15:03:52.728214 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.716815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33269 (local address 127.2.164.67:57105)
0202 15:03:52.717376 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:52.717424 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:52.718242 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:52.718722 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:52.718729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:52.719105 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:52.720037 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:52.720050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.720779 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.720781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:52.721258 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:52.721266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.721465 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.722166 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:52.722191 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:52.723634 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:52.725478 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:52.725483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:52.725493 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:52.725745 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:52.726049 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:52.726052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:52.726054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:52.726157 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:52.726567 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:52.726573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:52.726821 (+ 248us) client_negotiation.cc:769] Sending connection context
0202 15:03:52.727015 (+ 194us) client_negotiation.cc:240] Negotiation successful
0202 15:03:52.727250 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":175,"threads_started":1}
I20260202 15:03:52.729349 7336 heartbeater.cc:344] Connected to a master server at 127.2.164.126:33269
I20260202 15:03:52.729601 7336 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:52.730144 7336 heartbeater.cc:507] Master 127.2.164.126:33269 requested a full tablet report, sending...
I20260202 15:03:52.731139 6871 ts_manager.cc:194] Registered new tserver with Master: 52c52033d6044764af816b89e21da2a5 (127.2.164.67:33693)
I20260202 15:03:52.731739 6871 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:57105
I20260202 15:03:52.741676 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:03:52.768077 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.758811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.2.164.65:46475)
0202 15:03:52.759154 (+ 343us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:52.759160 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:52.759187 (+ 27us) server_negotiation.cc:408] Connection header received
0202 15:03:52.759242 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:52.759247 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:52.759425 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:52.759542 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:52.760558 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.761433 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.762138 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.762337 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.764370 (+ 2033us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:52.764401 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:52.764422 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:52.764450 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:52.766219 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.766674 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.766679 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.766680 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.766755 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.767042 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.767046 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.767047 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.767290 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:52.767475 (+ 185us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:52.767693 (+ 218us) server_negotiation.cc:300] Negotiation successful
0202 15:03:52.767883 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":109,"threads_started":1}
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[6823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:52.779042 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:52.770920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.2.164.126:33269)
0202 15:03:52.771059 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:52.771063 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:52.771076 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:03:52.771184 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:52.771187 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:52.771230 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:52.771299 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:52.771981 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.772492 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:52.773263 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:52.773408 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:52.775696 (+ 2288us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:52.775717 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:52.775720 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:52.775749 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:52.777418 (+ 1669us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.777807 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.777810 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.777811 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.777853 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:52.778086 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:52.778089 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:52.778090 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:52.778307 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:52.778429 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:52.778687 (+ 258us) server_negotiation.cc:300] Negotiation successful
0202 15:03:52.778789 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260202 15:03:52.779481 6871 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47130
I20260202 15:03:52.780179 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6935
I20260202 15:03:52.786062 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7071
I20260202 15:03:52.792040 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7207
I20260202 15:03:52.797906 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 6839
2026-02-02T15:03:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3187 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[7348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[7348](info): set up 2 sockets
Feb 02 15:03:52 dist-test-slave-6zzp krb5kdc[7348](info): commencing operation
krb5kdc: starting...
W20260202 15:03:54.813391 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.992s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:03:54 dist-test-slave-6zzp krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:03:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:54.952455 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33545
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:55.059019 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:55.059319 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:55.059412 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:55.062638 7364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:55.062754 7364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:55.062803 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:55.062865 7364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:55.062913 7364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:55.067114 7364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:55.068250 7364 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:55.069166 7364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:55.074821 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.074913 7364 server_base.cc:1060] running on GCE node
W20260202 15:03:55.074828 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:55.075101 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.075591 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:55.076471 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:55.077621 7364 hybrid_clock.cc:648] HybridClock initialized: now 1770044635077609 us; error 30 us; skew 500 ppm
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:55.080157 7364 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:03:55.080880 7364 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:55.080937 7364 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:55.081152 7364 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:55.082834 7364 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "fba071e8daf940b99dfd25525a90484d"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "2cd16fc2ae69e21317172c7a941f2aa7"
server_key_iv: "1262fb51806a99cacbd8a5a7ac4c65ee"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.083317 7364 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "fba071e8daf940b99dfd25525a90484d"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "2cd16fc2ae69e21317172c7a941f2aa7"
server_key_iv: "1262fb51806a99cacbd8a5a7ac4c65ee"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.086551 7364 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:03:55.088666 7378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.089804 7364 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:55.089944 7364 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "fba071e8daf940b99dfd25525a90484d"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "2cd16fc2ae69e21317172c7a941f2aa7"
server_key_iv: "1262fb51806a99cacbd8a5a7ac4c65ee"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.090047 7364 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:55.108701 7364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:55.111399 7364 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:55.111600 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:55.118927 7364 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:33545
I20260202 15:03:55.118948 7430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:33545 every 8 connection(s)
I20260202 15:03:55.119612 7364 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:55.122363 7431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:55.127627 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: Bootstrap starting.
I20260202 15:03:55.128052 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7364
I20260202 15:03:55.128206 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:55.128468 2705 external_mini_cluster.cc:1468] Setting key 06fb45e88443c8393d3d0650be35008d
I20260202 15:03:55.130149 7431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:55.130851 7431 log.cc:826] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:55.132581 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: No bootstrap required, opened a new log
I20260202 15:03:55.135293 7431 raft_consensus.cc:359] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba071e8daf940b99dfd25525a90484d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33545 } }
I20260202 15:03:55.135535 7431 raft_consensus.cc:385] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:55.135601 7431 raft_consensus.cc:740] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba071e8daf940b99dfd25525a90484d, State: Initialized, Role: FOLLOWER
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:55.136070 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba071e8daf940b99dfd25525a90484d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33545 } }
I20260202 15:03:55.136188 7431 raft_consensus.cc:399] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:55.136276 7431 raft_consensus.cc:493] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:55.136385 7431 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:55.137490 7431 raft_consensus.cc:515] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba071e8daf940b99dfd25525a90484d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33545 } }
I20260202 15:03:55.137858 7431 leader_election.cc:304] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba071e8daf940b99dfd25525a90484d; no voters:
I20260202 15:03:55.138146 7431 leader_election.cc:290] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:55.138290 7436 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:55.138510 7436 raft_consensus.cc:697] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [term 1 LEADER]: Becoming Leader. State: Replica: fba071e8daf940b99dfd25525a90484d, State: Running, Role: LEADER
I20260202 15:03:55.138826 7436 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba071e8daf940b99dfd25525a90484d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33545 } }
I20260202 15:03:55.139482 7431 sys_catalog.cc:565] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:55.140674 7437 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba071e8daf940b99dfd25525a90484d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba071e8daf940b99dfd25525a90484d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33545 } } }
I20260202 15:03:55.140869 7437 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:55.140157 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba071e8daf940b99dfd25525a90484d. Latest consensus state: current_term: 1 leader_uuid: "fba071e8daf940b99dfd25525a90484d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba071e8daf940b99dfd25525a90484d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33545 } } }
I20260202 15:03:55.141554 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:55.142481 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.129638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.2.164.126:33545)
0202 15:03:55.130060 (+ 422us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.130070 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.130104 (+ 34us) server_negotiation.cc:408] Connection header received
0202 15:03:55.130684 (+ 580us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.130700 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.131026 (+ 326us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.131368 (+ 342us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:55.132272 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.133119 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.133792 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.134084 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.136496 (+ 2412us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:55.136520 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:55.136533 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:55.136574 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:55.139349 (+ 2775us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.139907 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.139913 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.139918 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.140011 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.140266 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.140270 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.140272 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.140709 (+ 437us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:55.140897 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.141401 (+ 504us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.141699 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
W20260202 15:03:55.144562 7451 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:03:55.144651 7451 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:03:55.145087 7445 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:55.145795 7445 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:55.150969 7445 catalog_manager.cc:1357] Generated new cluster ID: 37df9ad8d0e24b349832e40a60f6f922
I20260202 15:03:55.151031 7445 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:55.171751 7445 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:55.172683 7445 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:55.180773 7445 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: Generated new TSK 0
I20260202 15:03:55.181535 7445 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:55.239703 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33545
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:55.351070 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:55.351323 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:55.351435 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:55.354841 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:55.354909 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:55.355055 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:55.359473 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:55.360625 7459 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:55.361537 7459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:55.368526 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.368677 7459 server_base.cc:1060] running on GCE node
W20260202 15:03:55.368526 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:55.368527 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.369356 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:55.369899 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:55.371054 7459 hybrid_clock.cc:648] HybridClock initialized: now 1770044635371036 us; error 43 us; skew 500 ppm
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:55.373807 7459 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:03:55.374543 7459 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:55.374631 7459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:55.374854 7459 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:55.376569 7459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "447f8acccfa84a5aa3c2309c363e7338"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "2e5df8bea9a466c642a5b05cc219ca98"
server_key_iv: "ecaf2859547bc8a590565c64f22699fd"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.377149 7459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "447f8acccfa84a5aa3c2309c363e7338"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "2e5df8bea9a466c642a5b05cc219ca98"
server_key_iv: "ecaf2859547bc8a590565c64f22699fd"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.380477 7459 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260202 15:03:55.382745 7473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.383773 7459 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:03:55.383908 7459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "447f8acccfa84a5aa3c2309c363e7338"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "2e5df8bea9a466c642a5b05cc219ca98"
server_key_iv: "ecaf2859547bc8a590565c64f22699fd"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.384007 7459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:55.407212 7459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:55.410137 7459 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:55.410457 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:55.411026 7459 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:55.411944 7459 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:55.412019 7459 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.412093 7459 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:55.412129 7459 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.422246 7459 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:34795
I20260202 15:03:55.422339 7586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:34795 every 8 connection(s)
I20260202 15:03:55.422884 7459 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:55.427068 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7459
I20260202 15:03:55.427219 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:55.427490 2705 external_mini_cluster.cc:1468] Setting key 0477d294838e4cec688f9a76e833e0b2
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:03:55.437207 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.424818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:35107 (local address 127.2.164.126:33545)
0202 15:03:55.424981 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.424985 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.425788 (+ 803us) server_negotiation.cc:408] Connection header received
0202 15:03:55.426698 (+ 910us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.426701 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.426766 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.426892 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:55.428846 (+ 1954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.429489 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.430456 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.430712 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.433393 (+ 2681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:55.433416 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:55.433418 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:55.433445 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:55.435011 (+ 1566us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.435604 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.435608 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.435610 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.435664 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.435974 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.435978 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.435980 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.436148 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:55.436253 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.436837 (+ 584us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.436967 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:03:55.437849 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.425140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33545 (local address 127.2.164.65:35107)
0202 15:03:55.425627 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.425670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.426489 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.427061 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.427069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.427561 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:55.428683 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.428701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.429628 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.429635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.430295 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.430308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.430535 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.431298 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:55.431325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:55.433216 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:55.435198 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:55.435204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:55.435220 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:55.435477 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:55.435765 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:55.435769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:55.435773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:55.435871 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:55.436268 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:55.436277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:55.436565 (+ 288us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.436805 (+ 240us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.437134 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":112,"threads_started":1}
I20260202 15:03:55.439062 7587 heartbeater.cc:344] Connected to a master server at 127.2.164.126:33545
I20260202 15:03:55.439342 7587 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:03:55.439898 7587 heartbeater.cc:507] Master 127.2.164.126:33545 requested a full tablet report, sending...
I20260202 15:03:55.441682 7395 ts_manager.cc:194] Registered new tserver with Master: 447f8acccfa84a5aa3c2309c363e7338 (127.2.164.65:34795)
I20260202 15:03:55.442857 7395 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:35107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:55.481073 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33545
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:55.588395 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:55.588600 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:55.588696 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:55.592023 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:55.592082 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:55.592226 7594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:55.596490 7594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:55.597616 7594 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:55.598459 7594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:55.605125 7599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:55.605221 7600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:55.605125 7602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.605360 7594 server_base.cc:1060] running on GCE node
I20260202 15:03:55.605779 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:55.606402 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:55.607555 7594 hybrid_clock.cc:648] HybridClock initialized: now 1770044635607534 us; error 36 us; skew 500 ppm
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:55.610193 7594 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:03:55.610913 7594 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:55.610997 7594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:55.611215 7594 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:55.612968 7594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "471fc556cdf14b62976e5c9434905907"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "4f67db7e965de988121207321557bdf7"
server_key_iv: "5ec332b41e31805ea92471212578eee1"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.613693 7594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "471fc556cdf14b62976e5c9434905907"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "4f67db7e965de988121207321557bdf7"
server_key_iv: "5ec332b41e31805ea92471212578eee1"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.617799 7594 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260202 15:03:55.620443 7608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.621665 7594 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:55.621799 7594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "471fc556cdf14b62976e5c9434905907"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "4f67db7e965de988121207321557bdf7"
server_key_iv: "5ec332b41e31805ea92471212578eee1"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.621922 7594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:55.639938 7594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:55.643296 7594 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:55.643510 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:55.644070 7594 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:55.644950 7594 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:55.645045 7594 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.645138 7594 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:55.645190 7594 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.654232 7594 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:40967
I20260202 15:03:55.654253 7721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:40967 every 8 connection(s)
I20260202 15:03:55.654812 7594 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:55.656970 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7594
I20260202 15:03:55.657155 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:55.657482 2705 external_mini_cluster.cc:1468] Setting key 654df154bc77c3a238382d183f7d97dd
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:03:55.667883 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.656485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:59061 (local address 127.2.164.126:33545)
0202 15:03:55.656641 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.656645 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.657371 (+ 726us) server_negotiation.cc:408] Connection header received
0202 15:03:55.658340 (+ 969us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.658343 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.658388 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.658472 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:55.660323 (+ 1851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.660851 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.661813 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.661961 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.664445 (+ 2484us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:55.664473 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:55.664476 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:55.664500 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:55.665939 (+ 1439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.666457 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.666462 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.666465 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.666531 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.666823 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.666826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.666827 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.666971 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:55.667053 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.667576 (+ 523us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.667720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:03:55.668562 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.656744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33545 (local address 127.2.164.66:59061)
0202 15:03:55.657220 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.657265 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.658119 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.658656 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.658668 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.659151 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:55.660099 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.660112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.660983 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.660989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.661681 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.661694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.661907 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.662615 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:55.662656 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:55.664293 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:55.666061 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:55.666067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:55.666078 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:55.666314 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:55.666638 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:55.666640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:55.666642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:55.666733 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:55.667061 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:55.667066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:55.667281 (+ 215us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.667539 (+ 258us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.667799 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:03:55.669778 7722 heartbeater.cc:344] Connected to a master server at 127.2.164.126:33545
I20260202 15:03:55.670034 7722 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:55.670604 7722 heartbeater.cc:507] Master 127.2.164.126:33545 requested a full tablet report, sending...
I20260202 15:03:55.671697 7395 ts_manager.cc:194] Registered new tserver with Master: 471fc556cdf14b62976e5c9434905907 (127.2.164.66:40967)
I20260202 15:03:55.672335 7395 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:59061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:03:55.707918 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33545
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:03:55.818113 7729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:55.818337 7729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:55.818397 7729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:55.821839 7729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:55.821905 7729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:55.822044 7729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:55.826224 7729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:55.827286 7729 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:55.828119 7729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:55.834458 7734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:55.834467 7737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.834619 7729 server_base.cc:1060] running on GCE node
W20260202 15:03:55.834553 7735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:55.835029 7729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:55.835558 7729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:55.836707 7729 hybrid_clock.cc:648] HybridClock initialized: now 1770044635836688 us; error 33 us; skew 500 ppm
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:03:55.839497 7729 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:03:55.840246 7729 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:55.840304 7729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:55.840503 7729 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:55.842340 7729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "dd919615938946af97de14052a0c855c"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "d5c3d0d2b7ece99410a93838300cbc4c"
server_key_iv: "bf60260cea821f14735123c308feeb29"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.842837 7729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "dd919615938946af97de14052a0c855c"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "d5c3d0d2b7ece99410a93838300cbc4c"
server_key_iv: "bf60260cea821f14735123c308feeb29"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.846237 7729 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:03:55.848384 7743 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.849511 7729 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:55.849637 7729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "dd919615938946af97de14052a0c855c"
format_stamp: "Formatted at 2026-02-02 15:03:55 on dist-test-slave-6zzp"
server_key: "d5c3d0d2b7ece99410a93838300cbc4c"
server_key_iv: "bf60260cea821f14735123c308feeb29"
server_key_version: "encryptionkey@0"
I20260202 15:03:55.849735 7729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:55.864621 7729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:55.867682 7729 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:55.867897 7729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:55.868458 7729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:55.869347 7729 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:55.869419 7729 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.869514 7729 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:55.869567 7729 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:55.880000 7729 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:42801
I20260202 15:03:55.880025 7856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:42801 every 8 connection(s)
I20260202 15:03:55.880610 7729 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:55.883932 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7729
I20260202 15:03:55.884063 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:55.884357 2705 external_mini_cluster.cc:1468] Setting key ffe9faf89dc6c3be3a8312121a269666
Feb 02 15:03:55 dist-test-slave-6zzp krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:03:55.892935 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.882446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:60187 (local address 127.2.164.126:33545)
0202 15:03:55.882592 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.882596 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.883241 (+ 645us) server_negotiation.cc:408] Connection header received
0202 15:03:55.884333 (+ 1092us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.884337 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.884388 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.884531 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:55.886649 (+ 2118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.887139 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.887765 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.887894 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.890093 (+ 2199us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:55.890109 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:55.890111 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:55.890135 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:55.891360 (+ 1225us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.891788 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.891791 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.891792 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.891831 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.892058 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.892060 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.892061 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.892219 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:55.892293 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.892731 (+ 438us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.892816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:03:55.893635 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.882640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33545 (local address 127.2.164.67:60187)
0202 15:03:55.883071 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.883106 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.884067 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.884835 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.884848 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.885416 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:55.886520 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.886535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.887246 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.887248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.887671 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.887678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.887828 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.888454 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:03:55.888502 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:55.889956 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:55.891473 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:55.891478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:55.891487 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:55.891703 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:55.891906 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:03:55.891908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:03:55.891909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:03:55.891986 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:03:55.892302 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:55.892307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:55.892529 (+ 222us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.892691 (+ 162us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.892886 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20260202 15:03:55.894743 7857 heartbeater.cc:344] Connected to a master server at 127.2.164.126:33545
I20260202 15:03:55.894989 7857 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:55.895453 7857 heartbeater.cc:507] Master 127.2.164.126:33545 requested a full tablet report, sending...
I20260202 15:03:55.896414 7395 ts_manager.cc:194] Registered new tserver with Master: dd919615938946af97de14052a0c855c (127.2.164.67:42801)
I20260202 15:03:55.897127 7395 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:60187
I20260202 15:03:55.898273 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260202 15:03:55.906934 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.900323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.2.164.126:33545)
0202 15:03:55.900509 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.900513 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.900654 (+ 141us) server_negotiation.cc:408] Connection header received
0202 15:03:55.900805 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.900809 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.900859 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.900950 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:55.901773 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.902248 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.902871 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.903071 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.904070 (+ 999us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:55.904087 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:55.904090 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:03:55.904120 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:55.905405 (+ 1285us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.905825 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.905829 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.905833 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.905879 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:03:55.906112 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:03:55.906115 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:03:55.906117 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:03:55.906299 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:55.906441 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.906655 (+ 214us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.906766 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260202 15:03:55.909897 7395 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:55.912304 7395 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:55.925565 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.921340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:34795 (local address 127.0.0.1:35792)
0202 15:03:55.922166 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.922179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.922265 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.922829 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.922831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.922844 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:55.923049 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.923055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.924368 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.924372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.925223 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.925233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.925315 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.925333 (+ 18us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.925378 (+ 45us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.925429 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":88,"threads_started":1}
I20260202 15:03:55.925563 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.920670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42801 (local address 127.0.0.1:51288)
0202 15:03:55.921561 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.921599 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.921740 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.922584 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.922589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.922611 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:55.922861 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.922869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.924217 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.924220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.925137 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.925146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.925260 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.925284 (+ 24us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.925335 (+ 51us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.925399 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":92,"threads_started":1}
I20260202 15:03:55.926075 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.2.164.66:40967)
0202 15:03:55.921888 (+ 343us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.921895 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.922182 (+ 287us) server_negotiation.cc:408] Connection header received
0202 15:03:55.922275 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.922283 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.922453 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.922591 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:55.923205 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.923894 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.925067 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.925620 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.925737 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.925797 (+ 60us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.925879 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260202 15:03:55.925469 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.920816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:40967 (local address 127.0.0.1:46562)
0202 15:03:55.921965 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.921977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.922074 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.922750 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.922753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.922770 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:55.923044 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.923050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.924014 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.924021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.924920 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.924931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.925067 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.925122 (+ 55us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.925215 (+ 93us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.925304 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":117,"threads_started":1}
I20260202 15:03:55.926708 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.920905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.2.164.67:42801)
0202 15:03:55.922126 (+ 1221us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.922132 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.922157 (+ 25us) server_negotiation.cc:408] Connection header received
0202 15:03:55.922252 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.922258 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.922422 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.922572 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:55.922984 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.924074 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.925636 (+ 1562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.926414 (+ 778us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.926503 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.926552 (+ 49us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.926609 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1081,"thread_start_us":120,"threads_started":1}
I20260202 15:03:55.926910 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.2.164.65:34795)
0202 15:03:55.921888 (+ 343us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.921895 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.922353 (+ 458us) server_negotiation.cc:408] Connection header received
0202 15:03:55.922433 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.922438 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.922585 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.922692 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:55.923168 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.924234 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.925561 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.926548 (+ 987us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.926643 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.926701 (+ 58us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.926780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260202 15:03:55.929170 7656 tablet_service.cc:1505] Processing CreateTablet for tablet aa2a28ca7f1342f28290d1f2abea483a (DEFAULT_TABLE table=test-table [id=b9288d2a429148b8a7e5b6b8d4e969fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:55.929219 7791 tablet_service.cc:1505] Processing CreateTablet for tablet aa2a28ca7f1342f28290d1f2abea483a (DEFAULT_TABLE table=test-table [id=b9288d2a429148b8a7e5b6b8d4e969fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:55.930049 7656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2a28ca7f1342f28290d1f2abea483a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:55.930052 7791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2a28ca7f1342f28290d1f2abea483a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:55.933952 7521 tablet_service.cc:1505] Processing CreateTablet for tablet aa2a28ca7f1342f28290d1f2abea483a (DEFAULT_TABLE table=test-table [id=b9288d2a429148b8a7e5b6b8d4e969fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:55.934819 7521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2a28ca7f1342f28290d1f2abea483a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:55.935710 7874 tablet_bootstrap.cc:492] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: Bootstrap starting.
I20260202 15:03:55.938347 7875 tablet_bootstrap.cc:492] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Bootstrap starting.
I20260202 15:03:55.938741 7874 tablet_bootstrap.cc:654] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:55.939616 7874 log.cc:826] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:55.940402 7875 tablet_bootstrap.cc:654] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:55.941255 7877 tablet_bootstrap.cc:492] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338: Bootstrap starting.
I20260202 15:03:55.941269 7875 log.cc:826] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:55.941941 7874 tablet_bootstrap.cc:492] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: No bootstrap required, opened a new log
I20260202 15:03:55.942193 7874 ts_tablet_manager.cc:1403] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260202 15:03:55.943233 7877 tablet_bootstrap.cc:654] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:55.943318 7875 tablet_bootstrap.cc:492] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: No bootstrap required, opened a new log
I20260202 15:03:55.943508 7875 ts_tablet_manager.cc:1403] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260202 15:03:55.943925 7877 log.cc:826] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:55.945420 7877 tablet_bootstrap.cc:492] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338: No bootstrap required, opened a new log
I20260202 15:03:55.945616 7877 ts_tablet_manager.cc:1403] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:03:55.946102 7874 raft_consensus.cc:359] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.946341 7874 raft_consensus.cc:385] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:55.946398 7874 raft_consensus.cc:740] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471fc556cdf14b62976e5c9434905907, State: Initialized, Role: FOLLOWER
I20260202 15:03:55.946852 7874 consensus_queue.cc:260] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.947141 7875 raft_consensus.cc:359] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.947335 7875 raft_consensus.cc:385] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:55.947393 7875 raft_consensus.cc:740] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd919615938946af97de14052a0c855c, State: Initialized, Role: FOLLOWER
I20260202 15:03:55.947701 7722 heartbeater.cc:499] Master 127.2.164.126:33545 was elected leader, sending a full tablet report...
I20260202 15:03:55.947870 7874 ts_tablet_manager.cc:1434] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260202 15:03:55.947811 7875 consensus_queue.cc:260] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.948378 7857 heartbeater.cc:499] Master 127.2.164.126:33545 was elected leader, sending a full tablet report...
I20260202 15:03:55.948318 7877 raft_consensus.cc:359] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.948489 7877 raft_consensus.cc:385] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:55.948534 7877 raft_consensus.cc:740] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447f8acccfa84a5aa3c2309c363e7338, State: Initialized, Role: FOLLOWER
I20260202 15:03:55.948629 7875 ts_tablet_manager.cc:1434] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260202 15:03:55.948899 7877 consensus_queue.cc:260] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.949728 7877 ts_tablet_manager.cc:1434] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:03:55.949853 7587 heartbeater.cc:499] Master 127.2.164.126:33545 was elected leader, sending a full tablet report...
I20260202 15:03:55.993157 7880 raft_consensus.cc:493] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:55.993350 7880 raft_consensus.cc:515] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:55.994485 7880 leader_election.cc:290] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd919615938946af97de14052a0c855c (127.2.164.67:42801), 447f8acccfa84a5aa3c2309c363e7338 (127.2.164.65:34795)
I20260202 15:03:55.997699 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.994721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:34795 (local address 127.2.164.66:36575)
0202 15:03:55.994852 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.994869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.994979 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.995422 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.995426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.995447 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:55.995716 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.995724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.996520 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.996525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.997234 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.997250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.997383 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.997402 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.997453 (+ 51us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.997519 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260202 15:03:55.998122 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.994992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:36575 (local address 127.2.164.65:34795)
0202 15:03:55.995137 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.995141 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.995158 (+ 17us) server_negotiation.cc:408] Connection header received
0202 15:03:55.995221 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.995225 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.995295 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.995383 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:55.995847 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.996387 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.997367 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.997846 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.997898 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.997958 (+ 60us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.998014 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260202 15:03:55.998335 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.994839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:50115 (local address 127.2.164.67:42801)
0202 15:03:55.995000 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:55.995006 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:55.995270 (+ 264us) server_negotiation.cc:408] Connection header received
0202 15:03:55.995333 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:55.995336 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:55.995422 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:55.995514 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:03:55.995948 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.996461 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.997600 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.998080 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.998118 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:55.998168 (+ 50us) server_negotiation.cc:300] Negotiation successful
0202 15:03:55.998233 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260202 15:03:55.998333 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:55.994721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42801 (local address 127.2.164.66:50115)
0202 15:03:55.995142 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:55.995156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:55.995262 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:55.995545 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:55.995547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:55.995563 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:03:55.995826 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.995833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.996576 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:55.996579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:55.997234 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:55.997244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:55.997368 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:55.997395 (+ 27us) client_negotiation.cc:769] Sending connection context
0202 15:03:55.997450 (+ 55us) client_negotiation.cc:240] Negotiation successful
0202 15:03:55.997519 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":57,"thread_start_us":140,"threads_started":1}
I20260202 15:03:55.999017 7541 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2a28ca7f1342f28290d1f2abea483a" candidate_uuid: "471fc556cdf14b62976e5c9434905907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447f8acccfa84a5aa3c2309c363e7338" is_pre_election: true
I20260202 15:03:55.999015 7811 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2a28ca7f1342f28290d1f2abea483a" candidate_uuid: "471fc556cdf14b62976e5c9434905907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd919615938946af97de14052a0c855c" is_pre_election: true
I20260202 15:03:55.999315 7541 raft_consensus.cc:2468] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 471fc556cdf14b62976e5c9434905907 in term 0.
I20260202 15:03:55.999315 7811 raft_consensus.cc:2468] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 471fc556cdf14b62976e5c9434905907 in term 0.
I20260202 15:03:55.999801 7610 leader_election.cc:304] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 447f8acccfa84a5aa3c2309c363e7338, 471fc556cdf14b62976e5c9434905907, dd919615938946af97de14052a0c855c; no voters:
I20260202 15:03:56.000077 7880 raft_consensus.cc:2804] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:56.000144 7880 raft_consensus.cc:493] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:56.000176 7880 raft_consensus.cc:3060] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.001165 7880 raft_consensus.cc:515] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:56.001536 7880 leader_election.cc:290] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [CANDIDATE]: Term 1 election: Requested vote from peers dd919615938946af97de14052a0c855c (127.2.164.67:42801), 447f8acccfa84a5aa3c2309c363e7338 (127.2.164.65:34795)
I20260202 15:03:56.001863 7811 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2a28ca7f1342f28290d1f2abea483a" candidate_uuid: "471fc556cdf14b62976e5c9434905907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd919615938946af97de14052a0c855c"
I20260202 15:03:56.001888 7541 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2a28ca7f1342f28290d1f2abea483a" candidate_uuid: "471fc556cdf14b62976e5c9434905907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447f8acccfa84a5aa3c2309c363e7338"
I20260202 15:03:56.001992 7811 raft_consensus.cc:3060] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.002002 7541 raft_consensus.cc:3060] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.002907 7811 raft_consensus.cc:2468] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471fc556cdf14b62976e5c9434905907 in term 1.
I20260202 15:03:56.002943 7541 raft_consensus.cc:2468] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471fc556cdf14b62976e5c9434905907 in term 1.
I20260202 15:03:56.003262 7612 leader_election.cc:304] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471fc556cdf14b62976e5c9434905907, dd919615938946af97de14052a0c855c; no voters:
I20260202 15:03:56.003433 7880 raft_consensus.cc:2804] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:56.003682 7880 raft_consensus.cc:697] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 1 LEADER]: Becoming Leader. State: Replica: 471fc556cdf14b62976e5c9434905907, State: Running, Role: LEADER
I20260202 15:03:56.004027 7880 consensus_queue.cc:237] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } }
I20260202 15:03:56.007026 7395 catalog_manager.cc:5654] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 reported cstate change: term changed from 0 to 1, leader changed from <none> to 471fc556cdf14b62976e5c9434905907 (127.2.164.66). New cstate: current_term: 1 leader_uuid: "471fc556cdf14b62976e5c9434905907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "471fc556cdf14b62976e5c9434905907" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 } health_report { overall_health: UNKNOWN } } }
I20260202 15:03:56.025187 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.022024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.2.164.66:40967)
0202 15:03:56.022162 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.022166 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.022181 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:03:56.022263 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.022266 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.022342 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.022460 (+ 118us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:03:56.022847 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:56.023365 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:56.024059 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:56.024213 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:56.024309 (+ 96us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:03:56.024766 (+ 457us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:03:56.024866 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.024950 (+ 84us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.025026 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:03:56.030702 7811 raft_consensus.cc:1275] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 1 FOLLOWER]: Refusing update from remote peer 471fc556cdf14b62976e5c9434905907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:56.030704 7541 raft_consensus.cc:1275] T aa2a28ca7f1342f28290d1f2abea483a P 447f8acccfa84a5aa3c2309c363e7338 [term 1 FOLLOWER]: Refusing update from remote peer 471fc556cdf14b62976e5c9434905907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:56.031373 7880 consensus_queue.cc:1048] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "447f8acccfa84a5aa3c2309c363e7338" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:56.031589 7884 consensus_queue.cc:1048] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd919615938946af97de14052a0c855c" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:56.049263 7395 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:56.049460 7395 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:56.051929 7395 catalog_manager.cc:5941] T 00000000000000000000000000000000 P fba071e8daf940b99dfd25525a90484d: Sending DeleteTablet for 3 replicas of tablet aa2a28ca7f1342f28290d1f2abea483a
I20260202 15:03:56.052886 7656 tablet_service.cc:1552] Processing DeleteTablet for tablet aa2a28ca7f1342f28290d1f2abea483a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:56 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:46562
I20260202 15:03:56.052925 7521 tablet_service.cc:1552] Processing DeleteTablet for tablet aa2a28ca7f1342f28290d1f2abea483a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:56 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:35792
I20260202 15:03:56.053171 7791 tablet_service.cc:1552] Processing DeleteTablet for tablet aa2a28ca7f1342f28290d1f2abea483a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:56 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:51288
I20260202 15:03:56.053663 7899 tablet_replica.cc:333] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: stopping tablet replica
I20260202 15:03:56.054049 7899 raft_consensus.cc:2243] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:56.054091 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7459
I20260202 15:03:56.054422 7899 raft_consensus.cc:2272] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:56.055948 7899 ts_tablet_manager.cc:1916] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:56.058264 7899 ts_tablet_manager.cc:1929] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:03:56.058363 7899 log.cc:1199] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/aa2a28ca7f1342f28290d1f2abea483a
I20260202 15:03:56.058724 7899 ts_tablet_manager.cc:1950] T aa2a28ca7f1342f28290d1f2abea483a P dd919615938946af97de14052a0c855c: Deleting consensus metadata
I20260202 15:03:56.059919 7382 catalog_manager.cc:4985] TS dd919615938946af97de14052a0c855c (127.2.164.67:42801): tablet aa2a28ca7f1342f28290d1f2abea483a (table test-table [id=b9288d2a429148b8a7e5b6b8d4e969fd]) successfully deleted
I20260202 15:03:56.061214 7898 tablet_replica.cc:333] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: stopping tablet replica
I20260202 15:03:56.061496 7898 raft_consensus.cc:2243] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:03:56.061918 7898 raft_consensus.cc:2272] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:56.063195 7898 ts_tablet_manager.cc:1916] T aa2a28ca7f1342f28290d1f2abea483a P 471fc556cdf14b62976e5c9434905907: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260202 15:03:56.063448 7380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:34795 (error 108)
I20260202 15:03:56.064280 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7594
I20260202 15:03:56.064865 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.064569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:34795 (local address 127.0.0.1:35796)
0202 15:03:56.064728 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.064778 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260202 15:03:56.065039 7380 catalog_manager.cc:4712] TS 447f8acccfa84a5aa3c2309c363e7338 (127.2.164.65:34795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa2a28ca7f1342f28290d1f2abea483a: Network error: Client connection negotiation failed: client connection to 127.2.164.65:34795: connect: Connection refused (error 111)
I20260202 15:03:56.070503 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7729
I20260202 15:03:56.071533 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.070763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:40967 (local address 127.0.0.1:46576)
0202 15:03:56.071371 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.071429 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.66:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470}
I20260202 15:03:56.076438 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7364
2026-02-02T15:03:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3278 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-02-02T15:03:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:56Z Disabled control of system clock
I20260202 15:03:56.106861 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39199
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260202 15:03:56.214196 7904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:56.214454 7904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:56.214504 7904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:56.217851 7904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:56.217913 7904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:56.217931 7904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:56.217949 7904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:56.217998 7904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:03:56.222059 7904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:39199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:56.223212 7904 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:56.224391 7904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:56.230233 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.230242 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.230253 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:56.230556 7904 server_base.cc:1060] running on GCE node
I20260202 15:03:56.231144 7904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:56.232129 7904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:56.233304 7904 hybrid_clock.cc:648] HybridClock initialized: now 1770044636233278 us; error 45 us; skew 500 ppm
I20260202 15:03:56.235179 7904 webserver.cc:492] Webserver started at http://127.2.164.126:44751/ using document root <none> and password file <none>
I20260202 15:03:56.235764 7904 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:56.235848 7904 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:56.236068 7904 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:56.237854 7904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "09a38f9bab594c87b100ef1b6c9e6875"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "157a1f6481affef524034f02cff2a734"
server_key_iv: "3ce75bb803468b84e12ec28efb191ce8"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.238346 7904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "09a38f9bab594c87b100ef1b6c9e6875"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "157a1f6481affef524034f02cff2a734"
server_key_iv: "3ce75bb803468b84e12ec28efb191ce8"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.241827 7904 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:56.244561 7918 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.245848 7904 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:03:56.246003 7904 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "09a38f9bab594c87b100ef1b6c9e6875"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "157a1f6481affef524034f02cff2a734"
server_key_iv: "3ce75bb803468b84e12ec28efb191ce8"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.246122 7904 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:56.251127 7904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:56.251749 7904 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:56.251953 7904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:56.259368 7904 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:39199
I20260202 15:03:56.259372 7970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:39199 every 8 connection(s)
I20260202 15:03:56.260407 7904 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:56.262333 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7904
I20260202 15:03:56.262460 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:56.262718 2705 external_mini_cluster.cc:1468] Setting key 3f50354eab85d4df0e296528e5d88d1e
I20260202 15:03:56.263646 7971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:56.268056 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.264072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.2.164.126:39199)
0202 15:03:56.264716 (+ 644us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.264725 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.264759 (+ 34us) server_negotiation.cc:408] Connection header received
0202 15:03:56.265446 (+ 687us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.265464 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.265771 (+ 307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.266036 (+ 265us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.266929 (+ 893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.266934 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.266943 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.266953 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.266992 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.267003 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.267113 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.267255 (+ 142us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.267515 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260202 15:03:56.269917 7971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875: Bootstrap starting.
I20260202 15:03:56.271901 7971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:56.272660 7971 log.cc:826] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:56.274421 7971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875: No bootstrap required, opened a new log
I20260202 15:03:56.276770 7971 raft_consensus.cc:359] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a38f9bab594c87b100ef1b6c9e6875" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39199 } }
I20260202 15:03:56.276955 7971 raft_consensus.cc:385] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:56.277070 7971 raft_consensus.cc:740] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a38f9bab594c87b100ef1b6c9e6875, State: Initialized, Role: FOLLOWER
I20260202 15:03:56.277542 7971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a38f9bab594c87b100ef1b6c9e6875" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39199 } }
I20260202 15:03:56.277662 7971 raft_consensus.cc:399] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:56.277711 7971 raft_consensus.cc:493] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:56.277781 7971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.278599 7971 raft_consensus.cc:515] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a38f9bab594c87b100ef1b6c9e6875" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39199 } }
I20260202 15:03:56.278964 7971 leader_election.cc:304] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a38f9bab594c87b100ef1b6c9e6875; no voters:
I20260202 15:03:56.279258 7971 leader_election.cc:290] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:56.279407 7976 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:56.279645 7976 raft_consensus.cc:697] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [term 1 LEADER]: Becoming Leader. State: Replica: 09a38f9bab594c87b100ef1b6c9e6875, State: Running, Role: LEADER
I20260202 15:03:56.279927 7976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a38f9bab594c87b100ef1b6c9e6875" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39199 } }
I20260202 15:03:56.280417 7971 sys_catalog.cc:565] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:56.281672 7977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09a38f9bab594c87b100ef1b6c9e6875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a38f9bab594c87b100ef1b6c9e6875" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39199 } } }
I20260202 15:03:56.281682 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a38f9bab594c87b100ef1b6c9e6875. Latest consensus state: current_term: 1 leader_uuid: "09a38f9bab594c87b100ef1b6c9e6875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a38f9bab594c87b100ef1b6c9e6875" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39199 } } }
I20260202 15:03:56.281859 7977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:56.281898 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875 [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:56.282737 7985 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:56.285283 7985 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:56.290633 7985 catalog_manager.cc:1357] Generated new cluster ID: b862cfe59cb74f5faac4030af2c76558
I20260202 15:03:56.290714 7985 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:56.319453 7985 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:56.319818 7985 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:56.328118 7985 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875: Generated new TSK 0
I20260202 15:03:56.329097 7985 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:03:56.339581 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39199
--builtin_ntp_servers=127.2.164.84:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260202 15:03:56.447072 7995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:56.447345 7995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:56.447427 7995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:56.450901 7995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:56.450992 7995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:56.451104 7995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:03:56.455061 7995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:56.456189 7995 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:56.457360 7995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:56.463855 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.463866 8003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.463937 8000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:56.464077 7995 server_base.cc:1060] running on GCE node
I20260202 15:03:56.464515 7995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:56.465147 7995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:56.466328 7995 hybrid_clock.cc:648] HybridClock initialized: now 1770044636466306 us; error 38 us; skew 500 ppm
I20260202 15:03:56.468181 7995 webserver.cc:492] Webserver started at http://127.2.164.65:41165/ using document root <none> and password file <none>
I20260202 15:03:56.468770 7995 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:56.468825 7995 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:56.469081 7995 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:56.470839 7995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "4a0f5344908a4e23afc8c59211d82e75"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "57d5514f07af7388af466581b43c40ab"
server_key_iv: "229574efe387c3338e4ffdb8221fb72e"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.471352 7995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "4a0f5344908a4e23afc8c59211d82e75"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "57d5514f07af7388af466581b43c40ab"
server_key_iv: "229574efe387c3338e4ffdb8221fb72e"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.475160 7995 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260202 15:03:56.477706 8009 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.478820 7995 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:03:56.478964 7995 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "4a0f5344908a4e23afc8c59211d82e75"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "57d5514f07af7388af466581b43c40ab"
server_key_iv: "229574efe387c3338e4ffdb8221fb72e"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.479075 7995 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:56.484747 7995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:56.485457 7995 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:56.485603 7995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:56.486068 7995 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:56.487030 7995 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:56.487107 7995 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.487170 7995 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:56.487223 7995 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.497499 7995 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:39093
I20260202 15:03:56.497550 8122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:39093 every 8 connection(s)
I20260202 15:03:56.498514 7995 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:03:56.505498 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 7995
I20260202 15:03:56.505606 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:03:56.505817 2705 external_mini_cluster.cc:1468] Setting key 7dff7b652d8559a2856c4fab9e166a81
I20260202 15:03:56.507961 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39199
--builtin_ntp_servers=127.2.164.84:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260202 15:03:56.508513 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.500471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:59829 (local address 127.2.164.126:39199)
0202 15:03:56.500651 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.500656 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.501605 (+ 949us) server_negotiation.cc:408] Connection header received
0202 15:03:56.502376 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.502379 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.502419 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.502508 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.507939 (+ 5431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.507943 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.507945 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.507957 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.507978 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.507986 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.508128 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.508361 (+ 233us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.508401 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260202 15:03:56.509677 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.500778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39199 (local address 127.2.164.65:59829)
0202 15:03:56.501454 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.501490 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.502196 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.502670 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.502677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.502760 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.507592 (+ 4832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.507595 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.507655 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.507742 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.508131 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.508143 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.508152 (+ 9us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.508404 (+ 252us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.508672 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":99,"threads_started":1}
I20260202 15:03:56.511350 8123 heartbeater.cc:344] Connected to a master server at 127.2.164.126:39199
I20260202 15:03:56.511662 8123 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:56.512264 8123 heartbeater.cc:507] Master 127.2.164.126:39199 requested a full tablet report, sending...
I20260202 15:03:56.513954 7935 ts_manager.cc:194] Registered new tserver with Master: 4a0f5344908a4e23afc8c59211d82e75 (127.2.164.65:39093)
W20260202 15:03:56.624531 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:56.624765 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:56.624850 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:56.629056 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:56.629159 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:56.629299 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:03:56.633293 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:56.634438 8126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:56.635581 8126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:56.643079 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.643090 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.643090 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:56.643496 8126 server_base.cc:1060] running on GCE node
I20260202 15:03:56.643882 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:56.644433 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:56.645612 8126 hybrid_clock.cc:648] HybridClock initialized: now 1770044636645589 us; error 36 us; skew 500 ppm
I20260202 15:03:56.647512 8126 webserver.cc:492] Webserver started at http://127.2.164.66:36629/ using document root <none> and password file <none>
I20260202 15:03:56.648088 8126 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:56.648144 8126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:56.648352 8126 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:56.650070 8126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "3b0bfda433eb47bfaf82c542e615a970"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "d040a84b10794ef058759a0aab3e374f"
server_key_iv: "be5d6b4e73e7c84ad50a2d1185aa63c0"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.650640 8126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "3b0bfda433eb47bfaf82c542e615a970"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "d040a84b10794ef058759a0aab3e374f"
server_key_iv: "be5d6b4e73e7c84ad50a2d1185aa63c0"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.653949 8126 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:03:56.656145 8140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.657449 8126 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:03:56.657582 8126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "3b0bfda433eb47bfaf82c542e615a970"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "d040a84b10794ef058759a0aab3e374f"
server_key_iv: "be5d6b4e73e7c84ad50a2d1185aa63c0"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.657693 8126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:56.663224 8126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:56.664022 8126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:56.664250 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:56.664791 8126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:56.665849 8126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:56.665897 8126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.665963 8126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:56.665999 8126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.675424 8126 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:45359
I20260202 15:03:56.675453 8253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:45359 every 8 connection(s)
I20260202 15:03:56.676424 8126 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:03:56.681470 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8126
I20260202 15:03:56.681625 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:03:56.681903 2705 external_mini_cluster.cc:1468] Setting key fa6a82613a5364da725fb02081141d65
I20260202 15:03:56.682060 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.678319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46483 (local address 127.2.164.126:39199)
0202 15:03:56.678482 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.678499 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.679454 (+ 955us) server_negotiation.cc:408] Connection header received
0202 15:03:56.680200 (+ 746us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.680203 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.680243 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.680320 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.681476 (+ 1156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.681480 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.681484 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.681499 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.681524 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.681536 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.681653 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.681864 (+ 211us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.681911 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260202 15:03:56.682934 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.678655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39199 (local address 127.2.164.66:46483)
0202 15:03:56.679306 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.679341 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.680032 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.680479 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.680486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.680568 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.681214 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.681217 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.681264 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.681329 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.681672 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.681682 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.681688 (+ 6us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.681892 (+ 204us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.682122 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260202 15:03:56.683984 8254 heartbeater.cc:344] Connected to a master server at 127.2.164.126:39199
I20260202 15:03:56.684008 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39199
--builtin_ntp_servers=127.2.164.84:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260202 15:03:56.684311 8254 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:56.684749 8254 heartbeater.cc:507] Master 127.2.164.126:39199 requested a full tablet report, sending...
I20260202 15:03:56.685891 7935 ts_manager.cc:194] Registered new tserver with Master: 3b0bfda433eb47bfaf82c542e615a970 (127.2.164.66:45359)
W20260202 15:03:56.795763 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:56.796030 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:56.796132 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:56.799612 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:56.799671 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:56.799804 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:03:56.803685 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:56.804800 8257 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:56.805987 8257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:56.812760 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:56.812819 8257 server_base.cc:1060] running on GCE node
W20260202 15:03:56.812764 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:56.812757 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:56.813500 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:56.814049 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:56.815245 8257 hybrid_clock.cc:648] HybridClock initialized: now 1770044636815227 us; error 48 us; skew 500 ppm
I20260202 15:03:56.816989 8257 webserver.cc:492] Webserver started at http://127.2.164.67:45015/ using document root <none> and password file <none>
I20260202 15:03:56.817621 8257 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:56.817703 8257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:56.817919 8257 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:56.819653 8257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "132a321b348f45658124852475071ea9"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "d63f37e28c6a1be196237b062e19a0fa"
server_key_iv: "be28cefbba1110abf49179b8f46902ac"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.820137 8257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "132a321b348f45658124852475071ea9"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "d63f37e28c6a1be196237b062e19a0fa"
server_key_iv: "be28cefbba1110abf49179b8f46902ac"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.823652 8257 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260202 15:03:56.826009 8271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.827051 8257 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260202 15:03:56.827189 8257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "132a321b348f45658124852475071ea9"
format_stamp: "Formatted at 2026-02-02 15:03:56 on dist-test-slave-6zzp"
server_key: "d63f37e28c6a1be196237b062e19a0fa"
server_key_iv: "be28cefbba1110abf49179b8f46902ac"
server_key_version: "encryptionkey@0"
I20260202 15:03:56.827328 8257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:56.832644 8257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:56.833397 8257 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:56.833602 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:56.834153 8257 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:03:56.835156 8257 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:03:56.835239 8257 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.835314 8257 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:03:56.835351 8257 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:56.844919 8257 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:44623
I20260202 15:03:56.845032 8384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:44623 every 8 connection(s)
I20260202 15:03:56.845917 8257 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:03:56.849993 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8257
I20260202 15:03:56.850126 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:03:56.850435 2705 external_mini_cluster.cc:1468] Setting key fc151dc8a64031cbbc09512c04338ad0
I20260202 15:03:56.851501 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.847682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:60219 (local address 127.2.164.126:39199)
0202 15:03:56.847895 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.847900 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.848716 (+ 816us) server_negotiation.cc:408] Connection header received
0202 15:03:56.849559 (+ 843us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.849562 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.849600 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.849675 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.850970 (+ 1295us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.850974 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.850978 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.850999 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.851020 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.851029 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.851133 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.851295 (+ 162us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.851344 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260202 15:03:56.852382 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.847963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39199 (local address 127.2.164.67:60219)
0202 15:03:56.848579 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.848615 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.849368 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.849837 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.849869 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.849969 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.850735 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.850737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.850769 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.850823 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.851122 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.851131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.851136 (+ 5us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.851362 (+ 226us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.851570 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260202 15:03:56.853456 8385 heartbeater.cc:344] Connected to a master server at 127.2.164.126:39199
I20260202 15:03:56.853667 8385 heartbeater.cc:461] Registering TS with master...
I20260202 15:03:56.854030 8385 heartbeater.cc:507] Master 127.2.164.126:39199 requested a full tablet report, sending...
I20260202 15:03:56.854961 7935 ts_manager.cc:194] Registered new tserver with Master: 132a321b348f45658124852475071ea9 (127.2.164.67:44623)
I20260202 15:03:56.864317 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260202 15:03:56.867928 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.866230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.2.164.126:39199)
0202 15:03:56.866396 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.866401 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.866550 (+ 149us) server_negotiation.cc:408] Connection header received
0202 15:03:56.866702 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.866706 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.866752 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.866835 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.867487 (+ 652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.867490 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.867493 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.867504 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.867524 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.867532 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.867627 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.867745 (+ 118us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.867783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:03:56.870481 7935 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:03:56.872913 7935 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:03:56.883903 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.881207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:45359 (local address 127.0.0.1:42302)
0202 15:03:56.881630 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.881670 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.881766 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.882283 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.882289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.882352 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.882958 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.882963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.883005 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.883069 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.883559 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.883571 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.883580 (+ 9us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.883674 (+ 94us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.883735 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":125,"threads_started":1}
I20260202 15:03:56.884498 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.881761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:39093 (local address 127.0.0.1:57610)
0202 15:03:56.881982 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.881996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.882608 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.883205 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.883208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.883226 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.883715 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.883717 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.883728 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.883746 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.884223 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.884225 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.884228 (+ 3us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.884326 (+ 98us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.884360 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260202 15:03:56.884577 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.881392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.2.164.66:45359)
0202 15:03:56.881806 (+ 414us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.881815 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.881841 (+ 26us) server_negotiation.cc:408] Connection header received
0202 15:03:56.881902 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.881908 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.882083 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.882631 (+ 548us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.883346 (+ 715us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.883355 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.883366 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.883380 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.883433 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.883447 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.884282 (+ 835us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.884331 (+ 49us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.884405 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":114,"threads_started":1}
I20260202 15:03:56.884653 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.882153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.2.164.65:39093)
0202 15:03:56.882684 (+ 531us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.882694 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.882720 (+ 26us) server_negotiation.cc:408] Connection header received
0202 15:03:56.882776 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.882784 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.882939 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.883070 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.883854 (+ 784us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.883859 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.883865 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.883875 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.883913 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.883921 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.884070 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.884341 (+ 271us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.884403 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":131,"threads_started":1}
I20260202 15:03:56.886431 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.881940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:44623 (local address 127.0.0.1:52606)
0202 15:03:56.883532 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.883547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.883606 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.884405 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.884409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.884422 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.884840 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.884842 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.884853 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.884876 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.886193 (+ 1317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.886199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.886202 (+ 3us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.886282 (+ 80us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.886315 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":75,"threads_started":1}
I20260202 15:03:56.886555 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.882316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.2.164.67:44623)
0202 15:03:56.884012 (+ 1696us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.884020 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.884044 (+ 24us) server_negotiation.cc:408] Connection header received
0202 15:03:56.884099 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.884105 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.884245 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.884745 (+ 500us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.886007 (+ 1262us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.886012 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.886020 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.886031 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.886072 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.886082 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.886201 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.886341 (+ 140us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.886404 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1577,"thread_start_us":146,"threads_started":1}
I20260202 15:03:56.887570 8188 tablet_service.cc:1505] Processing CreateTablet for tablet ca852caa93b74c04b691b5df4a474655 (DEFAULT_TABLE table=test-table [id=3d5aa3f300634a908c25b7a6fb288795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:56.887570 8057 tablet_service.cc:1505] Processing CreateTablet for tablet ca852caa93b74c04b691b5df4a474655 (DEFAULT_TABLE table=test-table [id=3d5aa3f300634a908c25b7a6fb288795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:56.888451 8188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca852caa93b74c04b691b5df4a474655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:56.888477 8057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca852caa93b74c04b691b5df4a474655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:56.888971 8319 tablet_service.cc:1505] Processing CreateTablet for tablet ca852caa93b74c04b691b5df4a474655 (DEFAULT_TABLE table=test-table [id=3d5aa3f300634a908c25b7a6fb288795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:03:56.889801 8319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca852caa93b74c04b691b5df4a474655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:56.893886 8402 tablet_bootstrap.cc:492] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Bootstrap starting.
I20260202 15:03:56.895257 8403 tablet_bootstrap.cc:492] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: Bootstrap starting.
I20260202 15:03:56.895531 8404 tablet_bootstrap.cc:492] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Bootstrap starting.
I20260202 15:03:56.896328 8402 tablet_bootstrap.cc:654] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:56.897244 8402 log.cc:826] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:56.897388 8403 tablet_bootstrap.cc:654] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:56.897420 8404 tablet_bootstrap.cc:654] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:56.898166 8403 log.cc:826] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:56.898181 8404 log.cc:826] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:56.899199 8402 tablet_bootstrap.cc:492] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: No bootstrap required, opened a new log
I20260202 15:03:56.899452 8402 ts_tablet_manager.cc:1403] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:03:56.899897 8403 tablet_bootstrap.cc:492] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: No bootstrap required, opened a new log
I20260202 15:03:56.899899 8404 tablet_bootstrap.cc:492] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: No bootstrap required, opened a new log
I20260202 15:03:56.900077 8404 ts_tablet_manager.cc:1403] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260202 15:03:56.900077 8403 ts_tablet_manager.cc:1403] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:03:56.901945 8402 raft_consensus.cc:359] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.902148 8402 raft_consensus.cc:385] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:56.902202 8402 raft_consensus.cc:740] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0bfda433eb47bfaf82c542e615a970, State: Initialized, Role: FOLLOWER
I20260202 15:03:56.902635 8402 consensus_queue.cc:260] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.903035 8404 raft_consensus.cc:359] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.903244 8404 raft_consensus.cc:385] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:56.903261 8254 heartbeater.cc:499] Master 127.2.164.126:39199 was elected leader, sending a full tablet report...
I20260202 15:03:56.903304 8404 raft_consensus.cc:740] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132a321b348f45658124852475071ea9, State: Initialized, Role: FOLLOWER
I20260202 15:03:56.903508 8402 ts_tablet_manager.cc:1434] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:03:56.903733 8404 consensus_queue.cc:260] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.903908 8403 raft_consensus.cc:359] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.904106 8403 raft_consensus.cc:385] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:56.904163 8403 raft_consensus.cc:740] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0f5344908a4e23afc8c59211d82e75, State: Initialized, Role: FOLLOWER
I20260202 15:03:56.904490 8385 heartbeater.cc:499] Master 127.2.164.126:39199 was elected leader, sending a full tablet report...
I20260202 15:03:56.904557 8403 consensus_queue.cc:260] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.904630 8404 ts_tablet_manager.cc:1434] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260202 15:03:56.905364 8403 ts_tablet_manager.cc:1434] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:03:56.905699 8123 heartbeater.cc:499] Master 127.2.164.126:39199 was elected leader, sending a full tablet report...
W20260202 15:03:56.927428 8255 tablet.cc:2378] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:03:56.965816 8410 raft_consensus.cc:493] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:03:56.966027 8410 raft_consensus.cc:515] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.967187 8410 leader_election.cc:290] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b0bfda433eb47bfaf82c542e615a970 (127.2.164.66:45359), 132a321b348f45658124852475071ea9 (127.2.164.67:44623)
I20260202 15:03:56.969321 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.967563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:45359 (local address 127.2.164.65:60359)
0202 15:03:56.967708 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.967726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.967843 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.968190 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.968193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.968207 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.968697 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.968699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.968731 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.968763 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.969096 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.969100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.969103 (+ 3us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.969185 (+ 82us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.969216 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260202 15:03:56.969396 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.967659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:60359 (local address 127.2.164.66:45359)
0202 15:03:56.967832 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.967838 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.967855 (+ 17us) server_negotiation.cc:408] Connection header received
0202 15:03:56.967942 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.967948 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.967993 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.968084 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.968911 (+ 827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.968914 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.968916 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.968944 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.968970 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.968981 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.969104 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.969188 (+ 84us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.969234 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260202 15:03:56.970010 8208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca852caa93b74c04b691b5df4a474655" candidate_uuid: "4a0f5344908a4e23afc8c59211d82e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0bfda433eb47bfaf82c542e615a970" is_pre_election: true
I20260202 15:03:56.970333 8208 raft_consensus.cc:2468] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a0f5344908a4e23afc8c59211d82e75 in term 0.
I20260202 15:03:56.970716 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.968000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34227 (local address 127.2.164.67:44623)
0202 15:03:56.968142 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:56.968146 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:56.968301 (+ 155us) server_negotiation.cc:408] Connection header received
0202 15:03:56.968408 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:56.968412 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:56.968451 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:56.968536 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:56.970252 (+ 1716us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:56.970257 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:56.970259 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:56.970273 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:56.970296 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:56.970307 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:56.970411 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:56.970559 (+ 148us) server_negotiation.cc:300] Negotiation successful
0202 15:03:56.970601 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260202 15:03:56.970736 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:56.967856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:44623 (local address 127.2.164.65:34227)
0202 15:03:56.968212 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:56.968223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:56.968324 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:56.968552 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:56.968555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:56.968568 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:03:56.969647 (+ 1079us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:03:56.969649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:03:56.969660 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:03:56.969682 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:03:56.970403 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:03:56.970406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:03:56.970425 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:03:56.970542 (+ 117us) client_negotiation.cc:240] Negotiation successful
0202 15:03:56.970577 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":153,"threads_started":1}
I20260202 15:03:56.970841 8010 leader_election.cc:304] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0bfda433eb47bfaf82c542e615a970, 4a0f5344908a4e23afc8c59211d82e75; no voters:
I20260202 15:03:56.971112 8410 raft_consensus.cc:2804] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:03:56.971251 8410 raft_consensus.cc:493] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:03:56.971318 8410 raft_consensus.cc:3060] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.971588 8339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca852caa93b74c04b691b5df4a474655" candidate_uuid: "4a0f5344908a4e23afc8c59211d82e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132a321b348f45658124852475071ea9" is_pre_election: true
I20260202 15:03:56.971834 8339 raft_consensus.cc:2468] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a0f5344908a4e23afc8c59211d82e75 in term 0.
I20260202 15:03:56.972513 8410 raft_consensus.cc:515] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.973119 8410 leader_election.cc:290] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [CANDIDATE]: Term 1 election: Requested vote from peers 3b0bfda433eb47bfaf82c542e615a970 (127.2.164.66:45359), 132a321b348f45658124852475071ea9 (127.2.164.67:44623)
I20260202 15:03:56.973469 8208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca852caa93b74c04b691b5df4a474655" candidate_uuid: "4a0f5344908a4e23afc8c59211d82e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0bfda433eb47bfaf82c542e615a970"
I20260202 15:03:56.973611 8339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca852caa93b74c04b691b5df4a474655" candidate_uuid: "4a0f5344908a4e23afc8c59211d82e75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132a321b348f45658124852475071ea9"
I20260202 15:03:56.973637 8208 raft_consensus.cc:3060] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.973721 8339 raft_consensus.cc:3060] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:56.975003 8339 raft_consensus.cc:2468] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a0f5344908a4e23afc8c59211d82e75 in term 1.
I20260202 15:03:56.975006 8208 raft_consensus.cc:2468] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a0f5344908a4e23afc8c59211d82e75 in term 1.
I20260202 15:03:56.975353 8010 leader_election.cc:304] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132a321b348f45658124852475071ea9, 4a0f5344908a4e23afc8c59211d82e75; no voters:
I20260202 15:03:56.975546 8410 raft_consensus.cc:2804] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:56.975766 8410 raft_consensus.cc:697] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [term 1 LEADER]: Becoming Leader. State: Replica: 4a0f5344908a4e23afc8c59211d82e75, State: Running, Role: LEADER
I20260202 15:03:56.976078 8410 consensus_queue.cc:237] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } }
I20260202 15:03:56.979743 7933 catalog_manager.cc:5654] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a0f5344908a4e23afc8c59211d82e75 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "4a0f5344908a4e23afc8c59211d82e75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a0f5344908a4e23afc8c59211d82e75" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39093 } health_report { overall_health: HEALTHY } } }
W20260202 15:03:56.999612 8124 tablet.cc:2378] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:03:57.009639 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:57.008083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.2.164.65:39093)
0202 15:03:57.008219 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:57.008224 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:57.008238 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:03:57.008345 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:57.008347 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:57.008404 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:57.008476 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:57.009195 (+ 719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:57.009198 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:57.009200 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:57.009211 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:57.009230 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:57.009238 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:57.009319 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:57.009408 (+ 89us) server_negotiation.cc:300] Negotiation successful
0202 15:03:57.009442 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:03:57.015352 8339 raft_consensus.cc:1275] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 1 FOLLOWER]: Refusing update from remote peer 4a0f5344908a4e23afc8c59211d82e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:57.015353 8208 raft_consensus.cc:1275] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 1 FOLLOWER]: Refusing update from remote peer 4a0f5344908a4e23afc8c59211d82e75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:03:57.016065 8410 consensus_queue.cc:1048] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0bfda433eb47bfaf82c542e615a970" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:57.016283 8412 consensus_queue.cc:1048] T ca852caa93b74c04b691b5df4a474655 P 4a0f5344908a4e23afc8c59211d82e75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "132a321b348f45658124852475071ea9" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:03:57.023238 8415 mvcc.cc:204] Tried to move back new op lower bound from 7250102833206763520 to 7250102833056743424. Current Snapshot: MvccSnapshot[applied={T|T < 7250102833206763520}]
I20260202 15:03:57.023842 8417 mvcc.cc:204] Tried to move back new op lower bound from 7250102833206763520 to 7250102833056743424. Current Snapshot: MvccSnapshot[applied={T|T < 7250102833206763520}]
I20260202 15:03:57.024605 8416 mvcc.cc:204] Tried to move back new op lower bound from 7250102833206763520 to 7250102833056743424. Current Snapshot: MvccSnapshot[applied={T|T < 7250102833206763520}]
I20260202 15:03:57.031574 7934 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:57.031708 7934 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:03:57.034173 7934 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 09a38f9bab594c87b100ef1b6c9e6875: Sending DeleteTablet for 3 replicas of tablet ca852caa93b74c04b691b5df4a474655
I20260202 15:03:57.034963 8057 tablet_service.cc:1552] Processing DeleteTablet for tablet ca852caa93b74c04b691b5df4a474655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:57 UTC) from {username='slave'} at 127.0.0.1:57610
I20260202 15:03:57.035001 8188 tablet_service.cc:1552] Processing DeleteTablet for tablet ca852caa93b74c04b691b5df4a474655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:57 UTC) from {username='slave'} at 127.0.0.1:42302
I20260202 15:03:57.035141 8319 tablet_service.cc:1552] Processing DeleteTablet for tablet ca852caa93b74c04b691b5df4a474655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:03:57 UTC) from {username='slave'} at 127.0.0.1:52606
I20260202 15:03:57.035583 8427 tablet_replica.cc:333] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: stopping tablet replica
I20260202 15:03:57.035880 8427 raft_consensus.cc:2243] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:57.035962 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7995
I20260202 15:03:57.036221 8427 raft_consensus.cc:2272] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:57.038442 8426 tablet_replica.cc:333] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: stopping tablet replica
I20260202 15:03:57.038722 8426 raft_consensus.cc:2243] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:03:57.039024 8426 raft_consensus.cc:2272] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:03:57.039021 8427 ts_tablet_manager.cc:1916] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:57.040474 8426 ts_tablet_manager.cc:1916] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:03:57.041843 8427 ts_tablet_manager.cc:1929] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:03:57.041955 8427 log.cc:1199] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/ca852caa93b74c04b691b5df4a474655
I20260202 15:03:57.042299 8427 ts_tablet_manager.cc:1950] T ca852caa93b74c04b691b5df4a474655 P 132a321b348f45658124852475071ea9: Deleting consensus metadata
I20260202 15:03:57.043103 8426 ts_tablet_manager.cc:1929] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:03:57.043200 8426 log.cc:1199] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/ca852caa93b74c04b691b5df4a474655
I20260202 15:03:57.043529 8426 ts_tablet_manager.cc:1950] T ca852caa93b74c04b691b5df4a474655 P 3b0bfda433eb47bfaf82c542e615a970: Deleting consensus metadata
I20260202 15:03:57.043752 7919 catalog_manager.cc:4985] TS 132a321b348f45658124852475071ea9 (127.2.164.67:44623): tablet ca852caa93b74c04b691b5df4a474655 (table test-table [id=3d5aa3f300634a908c25b7a6fb288795]) successfully deleted
W20260202 15:03:57.044381 7920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:39093 (error 108)
I20260202 15:03:57.044654 7919 catalog_manager.cc:4985] TS 3b0bfda433eb47bfaf82c542e615a970 (127.2.164.66:45359): tablet ca852caa93b74c04b691b5df4a474655 (table test-table [id=3d5aa3f300634a908c25b7a6fb288795]) successfully deleted
I20260202 15:03:57.045266 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8126
I20260202 15:03:57.045570 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:57.045237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:39093 (local address 127.0.0.1:57626)
0202 15:03:57.045390 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:57.045467 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:39093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260202 15:03:57.046007 7920 catalog_manager.cc:4712] TS 4a0f5344908a4e23afc8c59211d82e75 (127.2.164.65:39093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca852caa93b74c04b691b5df4a474655: Network error: Client connection negotiation failed: client connection to 127.2.164.65:39093: connect: Connection refused (error 111)
I20260202 15:03:57.051424 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8257
I20260202 15:03:57.056687 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 7904
2026-02-02T15:03:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (979 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260202 15:03:57.078857 2705 mini_oidc.cc:150] Starting JWKS server
I20260202 15:03:57.080235 2705 webserver.cc:492] Webserver started at https://127.0.0.1:44111/ using document root <none> and password file <none>
I20260202 15:03:57.080353 2705 mini_oidc.cc:186] Starting OIDC Discovery server
I20260202 15:03:57.080557 2705 webserver.cc:492] Webserver started at http://127.0.0.1:32911/ using document root <none> and password file <none>
2026-02-02T15:03:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:03:57Z Disabled control of system clock
I20260202 15:03:57.088315 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36245
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36245
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/testchainca.cert with env {}
W20260202 15:03:57.194063 8434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:03:57.194356 8434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:03:57.194445 8434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:03:57.197711 8434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:03:57.197801 8434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:03:57.197846 8434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:03:57.197892 8434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:03:57.197950 8434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260202 15:03:57.197991 8434 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260202 15:03:57.198022 8434 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44111/jwks.json
I20260202 15:03:57.202333 8434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:03:57.203403 8434 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:03:57.204232 8434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:03:57.209446 8439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:57.209446 8440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:03:57.209503 8442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:03:57.210180 8434 server_base.cc:1060] running on GCE node
I20260202 15:03:57.210678 8434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:03:57.211532 8434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:03:57.212707 8434 hybrid_clock.cc:648] HybridClock initialized: now 1770044637212672 us; error 46 us; skew 500 ppm
I20260202 15:03:57.214427 8434 webserver.cc:492] Webserver started at http://127.2.164.126:42193/ using document root <none> and password file <none>
I20260202 15:03:57.214995 8434 fs_manager.cc:362] Metadata directory not provided
I20260202 15:03:57.215049 8434 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:03:57.215242 8434 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:03:57.216835 8434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "f3c3fbd264af4b268b823a5b3023ecec"
format_stamp: "Formatted at 2026-02-02 15:03:57 on dist-test-slave-6zzp"
server_key: "78dc00aaddfcf73ebef3d9cd985e7e47"
server_key_iv: "e95f2c7d0b1e07801bcb00d285f64364"
server_key_version: "encryptionkey@0"
I20260202 15:03:57.217357 8434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "f3c3fbd264af4b268b823a5b3023ecec"
format_stamp: "Formatted at 2026-02-02 15:03:57 on dist-test-slave-6zzp"
server_key: "78dc00aaddfcf73ebef3d9cd985e7e47"
server_key_iv: "e95f2c7d0b1e07801bcb00d285f64364"
server_key_version: "encryptionkey@0"
I20260202 15:03:57.220397 8434 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:03:57.222504 8448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:03:57.223707 8434 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:03:57.223838 8434 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "f3c3fbd264af4b268b823a5b3023ecec"
format_stamp: "Formatted at 2026-02-02 15:03:57 on dist-test-slave-6zzp"
server_key: "78dc00aaddfcf73ebef3d9cd985e7e47"
server_key_iv: "e95f2c7d0b1e07801bcb00d285f64364"
server_key_version: "encryptionkey@0"
I20260202 15:03:57.223933 8434 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:03:57.235738 8434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:03:57.236291 8434 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:03:57.236474 8434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:03:57.243676 8434 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36245
I20260202 15:03:57.243687 8500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:36245 every 8 connection(s)
I20260202 15:03:57.244656 8434 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:03:57.247397 8501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:03:57.252642 8501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec: Bootstrap starting.
I20260202 15:03:57.253584 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8434
I20260202 15:03:57.253918 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:03:57.254207 2705 external_mini_cluster.cc:1468] Setting key 52f62a80f7d6dd1494d9f3e7b274546d
I20260202 15:03:57.255491 8501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec: Neither blocks nor log segments found. Creating new log.
I20260202 15:03:57.256201 8501 log.cc:826] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec: Log is configured to *not* fsync() on all Append() calls
I20260202 15:03:57.258278 8501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec: No bootstrap required, opened a new log
I20260202 15:03:57.261287 8501 raft_consensus.cc:359] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3fbd264af4b268b823a5b3023ecec" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36245 } }
I20260202 15:03:57.261495 8501 raft_consensus.cc:385] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:03:57.261586 8501 raft_consensus.cc:740] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c3fbd264af4b268b823a5b3023ecec, State: Initialized, Role: FOLLOWER
I20260202 15:03:57.262017 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:57.255244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.2.164.126:36245)
0202 15:03:57.255866 (+ 622us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:57.255875 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:57.255907 (+ 32us) server_negotiation.cc:408] Connection header received
0202 15:03:57.256505 (+ 598us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:57.256523 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:57.256904 (+ 381us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:57.257273 (+ 369us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:03:57.257778 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.258608 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:57.259435 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.259688 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:57.260224 (+ 536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:03:57.260243 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:03:57.260252 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:03:57.260631 (+ 379us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:03:57.260673 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:03:57.260687 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:03:57.260769 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:57.260964 (+ 195us) server_negotiation.cc:300] Negotiation successful
0202 15:03:57.261287 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":156,"threads_started":1}
I20260202 15:03:57.262008 8501 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3fbd264af4b268b823a5b3023ecec" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36245 } }
I20260202 15:03:57.262166 8501 raft_consensus.cc:399] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:03:57.262259 8501 raft_consensus.cc:493] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:03:57.262373 8501 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:03:57.263151 8501 raft_consensus.cc:515] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3fbd264af4b268b823a5b3023ecec" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36245 } }
I20260202 15:03:57.263489 8501 leader_election.cc:304] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c3fbd264af4b268b823a5b3023ecec; no voters:
I20260202 15:03:57.263849 8501 leader_election.cc:290] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:03:57.263945 8506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:03:57.264106 8506 raft_consensus.cc:697] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [term 1 LEADER]: Becoming Leader. State: Replica: f3c3fbd264af4b268b823a5b3023ecec, State: Running, Role: LEADER
I20260202 15:03:57.264369 8506 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3fbd264af4b268b823a5b3023ecec" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36245 } }
I20260202 15:03:57.264731 2705 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260202 15:03:57.264978 8501 sys_catalog.cc:565] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:03:57.265481 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c3fbd264af4b268b823a5b3023ecec. Latest consensus state: current_term: 1 leader_uuid: "f3c3fbd264af4b268b823a5b3023ecec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3fbd264af4b268b823a5b3023ecec" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36245 } } }
I20260202 15:03:57.265581 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:57.266225 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3c3fbd264af4b268b823a5b3023ecec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c3fbd264af4b268b823a5b3023ecec" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36245 } } }
I20260202 15:03:57.266320 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec [sys.catalog]: This master's current role is: LEADER
I20260202 15:03:57.266485 8510 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:03:57.269475 8510 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:03:57.274955 8510 catalog_manager.cc:1357] Generated new cluster ID: 6e407f7d54ed488aa5451f0646ae7b72
I20260202 15:03:57.275033 8510 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:03:57.292352 8510 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:03:57.293282 8510 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:03:57.301296 8510 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f3c3fbd264af4b268b823a5b3023ecec: Generated new TSK 0
I20260202 15:03:57.301949 8510 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:03:57.342104 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:57.312025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.2.164.126:36245)
0202 15:03:57.312152 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:57.312156 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:57.312350 (+ 194us) server_negotiation.cc:408] Connection header received
0202 15:03:57.312578 (+ 228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:57.312582 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:57.312640 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:57.312737 (+ 97us) server_negotiation.cc:227] Negotiated authn=JWT
0202 15:03:57.313171 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.313674 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:57.314325 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.314523 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:57.314606 (+ 83us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0202 15:03:57.341547 (+ 26941us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0202 15:03:57.341699 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0202 15:03:57.341859 (+ 160us) server_negotiation.cc:300] Negotiation successful
0202 15:03:57.341926 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"thread_start_us":287,"threads_started":1}
I20260202 15:03:57.352458 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:03:57.348835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.2.164.126:36245)
0202 15:03:57.348964 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0202 15:03:57.348968 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:03:57.349161 (+ 193us) server_negotiation.cc:408] Connection header received
0202 15:03:57.349276 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:03:57.349279 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:03:57.349323 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:03:57.349392 (+ 69us) server_negotiation.cc:227] Negotiated authn=JWT
0202 15:03:57.349867 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.350385 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:03:57.351261 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.351399 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:57.351437 (+ 38us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0202 15:03:57.352212 (+ 775us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0202 15:03:57.352373 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56932: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":32}
W20260202 15:03:57.352560 8504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56932: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260202 15:03:57.353367 8537 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:03:57.348733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36245 (local address 127.0.0.1:56932)
0202 15:03:57.349051 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0202 15:03:57.349067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:03:57.349176 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:03:57.349425 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:03:57.349428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:03:57.349436 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0202 15:03:57.349717 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:57.349725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.350513 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:03:57.350517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:03:57.351149 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:03:57.351159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:03:57.351281 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:03:57.351303 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0202 15:03:57.352388 (+ 1085us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0202 15:03:57.352542 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:36245: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
I20260202 15:04:12.360906 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:12.357762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.2.164.126:36245)
0202 15:04:12.358015 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.358018 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.358037 (+ 19us) server_negotiation.cc:408] Connection header received
0202 15:04:12.358098 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.358101 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.358149 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.358251 (+ 102us) server_negotiation.cc:227] Negotiated authn=JWT
0202 15:04:12.358652 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.359187 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.359818 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.359963 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.360007 (+ 44us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0202 15:04:12.360636 (+ 629us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0202 15:04:12.360782 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46584: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
W20260202 15:04:12.361016 8543 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46584: JWT verification failed: token expired
W20260202 15:04:12.361079 8542 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:12.357589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36245 (local address 127.0.0.1:46584)
0202 15:04:12.357910 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:12.357923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:12.358012 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:12.358276 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:12.358279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:12.358285 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0202 15:04:12.358487 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:12.358494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.359298 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.359301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:12.359704 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:12.359711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.359803 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.359824 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0202 15:04:12.360751 (+ 927us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0202 15:04:12.360865 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:36245: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
W20260202 15:04:12.363534 8543 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:12.362780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.2.164.126:36245)
0202 15:04:12.362929 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.362935 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.362951 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:04:12.363100 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.363105 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.363155 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.363248 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:12.363433 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46598: BlockingRecv error: recv got EOF from 127.0.0.1:46598 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260202 15:04:12.367307 8553 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.2.164.126:36245
W20260202 15:04:12.367878 8543 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:12.366996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.2.164.126:36245)
0202 15:04:12.367130 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.367134 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.367237 (+ 103us) server_negotiation.cc:408] Connection header received
0202 15:04:12.367473 (+ 236us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.367477 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.367527 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.367615 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:12.367758 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46612: BlockingRecv error: recv got EOF from 127.0.0.1:46612 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260202 15:04:12.368072 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8434
2026-02-02T15:04:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260202 15:04:12.408514 2705 mini_oidc.cc:150] Starting JWKS server
I20260202 15:04:12.409492 2705 webserver.cc:492] Webserver started at https://127.0.0.1:34567/ using document root <none> and password file <none>
I20260202 15:04:12.409564 2705 mini_oidc.cc:186] Starting OIDC Discovery server
I20260202 15:04:12.409792 2705 webserver.cc:492] Webserver started at http://127.0.0.1:45435/ using document root <none> and password file <none>
2026-02-02T15:04:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:12Z Disabled control of system clock
I20260202 15:04:12.417760 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40485
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34567/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/testchainca.cert with env {}
W20260202 15:04:12.527321 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:12.527587 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:12.527649 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:12.530933 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:12.531004 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:12.531023 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:12.531041 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:12.531059 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260202 15:04:12.531077 8559 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260202 15:04:12.531092 8559 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34567/jwks.json
I20260202 15:04:12.535666 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34567/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:12.536723 8559 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:12.537576 8559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:12.544358 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:12.544559 8559 server_base.cc:1060] running on GCE node
W20260202 15:04:12.544660 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:12.544838 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:12.545571 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:12.546929 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:12.548132 8559 hybrid_clock.cc:648] HybridClock initialized: now 1770044652548109 us; error 39 us; skew 500 ppm
I20260202 15:04:12.550482 8559 webserver.cc:492] Webserver started at http://127.2.164.126:34985/ using document root <none> and password file <none>
I20260202 15:04:12.551329 8559 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:12.551436 8559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:12.551884 8559 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:12.554293 8559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "5c1feccdbde248158956e6c2b88a8f04"
format_stamp: "Formatted at 2026-02-02 15:04:12 on dist-test-slave-6zzp"
server_key: "8a70aff273c50ec34b2a71dfec814d50"
server_key_iv: "38898e6b0780592b9afdbf357912d76e"
server_key_version: "encryptionkey@0"
I20260202 15:04:12.554883 8559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "5c1feccdbde248158956e6c2b88a8f04"
format_stamp: "Formatted at 2026-02-02 15:04:12 on dist-test-slave-6zzp"
server_key: "8a70aff273c50ec34b2a71dfec814d50"
server_key_iv: "38898e6b0780592b9afdbf357912d76e"
server_key_version: "encryptionkey@0"
I20260202 15:04:12.558454 8559 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:12.561486 8573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:12.562852 8559 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:04:12.562987 8559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "5c1feccdbde248158956e6c2b88a8f04"
format_stamp: "Formatted at 2026-02-02 15:04:12 on dist-test-slave-6zzp"
server_key: "8a70aff273c50ec34b2a71dfec814d50"
server_key_iv: "38898e6b0780592b9afdbf357912d76e"
server_key_version: "encryptionkey@0"
I20260202 15:04:12.563093 8559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:12.584555 8559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:12.585350 8559 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:12.585521 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:12.592684 8559 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40485
I20260202 15:04:12.592687 8625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:40485 every 8 connection(s)
I20260202 15:04:12.593883 8559 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:12.596721 8626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:12.602341 8626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04: Bootstrap starting.
I20260202 15:04:12.603166 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8559
I20260202 15:04:12.603283 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:12.603549 2705 external_mini_cluster.cc:1468] Setting key a05a85d859ef24e961005bf5c6ab677a
I20260202 15:04:12.605177 8626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:12.605902 8626 log.cc:826] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:12.607693 8626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04: No bootstrap required, opened a new log
I20260202 15:04:12.610535 8626 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feccdbde248158956e6c2b88a8f04" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40485 } }
I20260202 15:04:12.610749 8626 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:12.610803 8626 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1feccdbde248158956e6c2b88a8f04, State: Initialized, Role: FOLLOWER
I20260202 15:04:12.611261 8626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feccdbde248158956e6c2b88a8f04" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40485 } }
I20260202 15:04:12.611413 8626 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:12.611488 8626 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:12.611557 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:12.604591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.2.164.126:40485)
0202 15:04:12.605331 (+ 740us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.605341 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.605377 (+ 36us) server_negotiation.cc:408] Connection header received
0202 15:04:12.605923 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.605939 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.606330 (+ 391us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.606682 (+ 352us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:12.607175 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.607972 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.608696 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.608990 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.609557 (+ 567us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:12.609581 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:12.609601 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:12.610262 (+ 661us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:12.610306 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:12.610323 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:12.610451 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:12.610661 (+ 210us) server_negotiation.cc:300] Negotiation successful
0202 15:04:12.610928 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":148,"threads_started":1}
I20260202 15:04:12.611601 8626 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:12.612668 8626 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feccdbde248158956e6c2b88a8f04" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40485 } }
I20260202 15:04:12.613065 8626 leader_election.cc:304] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c1feccdbde248158956e6c2b88a8f04; no voters:
I20260202 15:04:12.613421 8626 leader_election.cc:290] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:12.613623 8631 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:12.613869 8631 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1feccdbde248158956e6c2b88a8f04, State: Running, Role: LEADER
I20260202 15:04:12.614172 8631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feccdbde248158956e6c2b88a8f04" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40485 } }
I20260202 15:04:12.614560 2705 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260202 15:04:12.614600 8626 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:12.616000 8632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c1feccdbde248158956e6c2b88a8f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feccdbde248158956e6c2b88a8f04" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40485 } } }
I20260202 15:04:12.616444 8632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:12.617112 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1feccdbde248158956e6c2b88a8f04. Latest consensus state: current_term: 1 leader_uuid: "5c1feccdbde248158956e6c2b88a8f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1feccdbde248158956e6c2b88a8f04" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40485 } } }
I20260202 15:04:12.617311 8645 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:12.617489 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:12.619906 8645 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:12.624857 8645 catalog_manager.cc:1357] Generated new cluster ID: e056b0b48b214b299df48fa245c68575
I20260202 15:04:12.624925 8645 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:12.642589 8645 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:12.643553 8645 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:12.649273 8645 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5c1feccdbde248158956e6c2b88a8f04: Generated new TSK 0
I20260202 15:04:12.650038 8645 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260202 15:04:12.677351 8656 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260202 15:04:12.677462 8656 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260202 15:04:12.678191 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:12.661188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.2.164.126:40485)
0202 15:04:12.661352 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.661358 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.661482 (+ 124us) server_negotiation.cc:408] Connection header received
0202 15:04:12.661646 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.661650 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.661719 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.661847 (+ 128us) server_negotiation.cc:227] Negotiated authn=JWT
0202 15:04:12.662178 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.663006 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.664019 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.664229 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.664357 (+ 128us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0202 15:04:12.677759 (+ 13402us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0202 15:04:12.678048 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260202 15:04:12.678268 8629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260202 15:04:12.678330 8655 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:12.661107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40485 (local address 127.0.0.1:34356)
0202 15:04:12.661395 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:12.661408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:12.661505 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:12.661840 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:12.661842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:12.661848 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0202 15:04:12.662071 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:12.662076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.663250 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.663256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:12.663885 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:12.663897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.664007 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.664028 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0202 15:04:12.677999 (+ 13971us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0202 15:04:12.678132 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:40485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
I20260202 15:04:12.678954 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8559
2026-02-02T15:04:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260202 15:04:12.719133 2705 mini_oidc.cc:150] Starting JWKS server
I20260202 15:04:12.720067 2705 webserver.cc:492] Webserver started at https://127.0.0.1:36605/ using document root <none> and password file <none>
I20260202 15:04:12.720197 2705 mini_oidc.cc:186] Starting OIDC Discovery server
I20260202 15:04:12.720454 2705 webserver.cc:492] Webserver started at http://127.0.0.1:33413/ using document root <none> and password file <none>
2026-02-02T15:04:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:12Z Disabled control of system clock
I20260202 15:04:12.728889 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38519
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38519
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260202 15:04:12.843631 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:12.844164 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:12.844300 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:12.849081 8662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:12.849220 8662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:12.849241 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:12.849263 8662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:12.849283 8662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260202 15:04:12.849335 8662 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260202 15:04:12.849380 8662 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36605/jwks.json
I20260202 15:04:12.854985 8662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:12.856329 8662 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:12.857669 8662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:12.863823 8667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:12.863927 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:12.863837 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:12.864944 8662 server_base.cc:1060] running on GCE node
I20260202 15:04:12.865685 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:12.866680 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:12.867856 8662 hybrid_clock.cc:648] HybridClock initialized: now 1770044652867833 us; error 43 us; skew 500 ppm
I20260202 15:04:12.869908 8662 webserver.cc:492] Webserver started at http://127.2.164.126:34837/ using document root <none> and password file <none>
I20260202 15:04:12.870525 8662 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:12.870600 8662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:12.870816 8662 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:12.872577 8662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "88d1f398e0b24f99baced1f846646839"
format_stamp: "Formatted at 2026-02-02 15:04:12 on dist-test-slave-6zzp"
server_key: "52758f641e604d375e2d804dd20f5f6d"
server_key_iv: "bc021959143259c42c4bc9d3e6a2170d"
server_key_version: "encryptionkey@0"
I20260202 15:04:12.873171 8662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "88d1f398e0b24f99baced1f846646839"
format_stamp: "Formatted at 2026-02-02 15:04:12 on dist-test-slave-6zzp"
server_key: "52758f641e604d375e2d804dd20f5f6d"
server_key_iv: "bc021959143259c42c4bc9d3e6a2170d"
server_key_version: "encryptionkey@0"
I20260202 15:04:12.876708 8662 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:12.879422 8676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:12.880694 8662 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260202 15:04:12.880856 8662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "88d1f398e0b24f99baced1f846646839"
format_stamp: "Formatted at 2026-02-02 15:04:12 on dist-test-slave-6zzp"
server_key: "52758f641e604d375e2d804dd20f5f6d"
server_key_iv: "bc021959143259c42c4bc9d3e6a2170d"
server_key_version: "encryptionkey@0"
I20260202 15:04:12.880987 8662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:12.894807 8662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:12.895507 8662 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:12.895705 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:12.903213 8662 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38519
I20260202 15:04:12.903209 8728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:38519 every 8 connection(s)
I20260202 15:04:12.904265 8662 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:12.904572 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8662
I20260202 15:04:12.904691 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:12.905037 2705 external_mini_cluster.cc:1468] Setting key 785fa54e344a671d7407aa67f8257547
I20260202 15:04:12.907496 8729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:12.913239 8729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839: Bootstrap starting.
I20260202 15:04:12.913938 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:12.906377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.2.164.126:38519)
0202 15:04:12.907264 (+ 887us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.907273 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.907309 (+ 36us) server_negotiation.cc:408] Connection header received
0202 15:04:12.908028 (+ 719us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.908047 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.908402 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.908698 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:12.909302 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.910460 (+ 1158us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.911372 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.911639 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.912214 (+ 575us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:12.912258 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:12.912268 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:12.912647 (+ 379us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:12.912691 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:12.912706 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:12.912796 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:12.913077 (+ 281us) server_negotiation.cc:300] Negotiation successful
0202 15:04:12.913321 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":394,"threads_started":1}
I20260202 15:04:12.915410 8729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:12.916225 8729 log.cc:826] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:12.916610 2705 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260202 15:04:12.918603 8729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839: No bootstrap required, opened a new log
I20260202 15:04:12.921193 8729 raft_consensus.cc:359] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d1f398e0b24f99baced1f846646839" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38519 } }
I20260202 15:04:12.921411 8729 raft_consensus.cc:385] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:12.921496 8729 raft_consensus.cc:740] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d1f398e0b24f99baced1f846646839, State: Initialized, Role: FOLLOWER
I20260202 15:04:12.921924 8729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d1f398e0b24f99baced1f846646839" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38519 } }
I20260202 15:04:12.922072 8729 raft_consensus.cc:399] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:12.922154 8729 raft_consensus.cc:493] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:12.922272 8729 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:12.923125 8729 raft_consensus.cc:515] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d1f398e0b24f99baced1f846646839" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38519 } }
I20260202 15:04:12.923472 8729 leader_election.cc:304] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d1f398e0b24f99baced1f846646839; no voters:
I20260202 15:04:12.923771 8729 leader_election.cc:290] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:12.923930 8734 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:12.924165 8734 raft_consensus.cc:697] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [term 1 LEADER]: Becoming Leader. State: Replica: 88d1f398e0b24f99baced1f846646839, State: Running, Role: LEADER
I20260202 15:04:12.924453 8734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d1f398e0b24f99baced1f846646839" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38519 } }
I20260202 15:04:12.925073 8729 sys_catalog.cc:565] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:12.926733 8735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d1f398e0b24f99baced1f846646839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d1f398e0b24f99baced1f846646839" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38519 } } }
I20260202 15:04:12.926879 8735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:12.927064 8736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d1f398e0b24f99baced1f846646839. Latest consensus state: current_term: 1 leader_uuid: "88d1f398e0b24f99baced1f846646839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d1f398e0b24f99baced1f846646839" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38519 } } }
I20260202 15:04:12.927182 8736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:12.927577 8741 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:12.930892 8741 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:12.936877 8741 catalog_manager.cc:1357] Generated new cluster ID: 3857785243ea4dc5b079b99d533d4e7c
I20260202 15:04:12.936975 8741 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:12.967886 8741 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:12.968883 8741 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:12.974911 8741 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 88d1f398e0b24f99baced1f846646839: Generated new TSK 0
I20260202 15:04:12.975775 8741 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260202 15:04:13.008129 8759 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260202 15:04:13.008212 8759 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260202 15:04:13.009538 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:12.988045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.2.164.126:38519)
0202 15:04:12.988200 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:12.988210 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:12.988364 (+ 154us) server_negotiation.cc:408] Connection header received
0202 15:04:12.988521 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:12.988524 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:12.988573 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:12.988653 (+ 80us) server_negotiation.cc:227] Negotiated authn=JWT
0202 15:04:12.989154 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.989970 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.990877 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.991019 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.991102 (+ 83us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0202 15:04:13.009138 (+ 18036us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0202 15:04:13.009428 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260202 15:04:13.009619 8732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260202 15:04:13.009999 8758 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:12.987970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38519 (local address 127.0.0.1:45498)
0202 15:04:12.988272 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:12.988290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:12.988402 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:12.988686 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:12.988689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:12.988697 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0202 15:04:12.989043 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:12.989053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.990110 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:12.990114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:12.990757 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:12.990767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:12.990892 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:12.990917 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0202 15:04:13.009351 (+ 18434us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0202 15:04:13.009536 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:38519: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260202 15:04:13.010601 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8662
2026-02-02T15:04:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-02-02T15:04:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:13Z Disabled control of system clock
I20260202 15:04:13.058789 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38475
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38475
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/testchainca.cert with env {}
W20260202 15:04:13.170893 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:13.171188 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:13.171280 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:13.174851 8763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:13.174976 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:13.175009 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:13.175053 8763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:13.175094 8763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260202 15:04:13.175136 8763 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260202 15:04:13.175170 8763 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260202 15:04:13.179626 8763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:13.180791 8763 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:13.181715 8763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:13.187652 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:13.187968 8763 server_base.cc:1060] running on GCE node
W20260202 15:04:13.187638 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:13.187623 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:13.188586 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:13.189577 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:13.190773 8763 hybrid_clock.cc:648] HybridClock initialized: now 1770044653190749 us; error 39 us; skew 500 ppm
I20260202 15:04:13.192533 8763 webserver.cc:492] Webserver started at http://127.2.164.126:39739/ using document root <none> and password file <none>
I20260202 15:04:13.193203 8763 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:13.193289 8763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:13.193513 8763 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:13.195251 8763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "ffb28890498b47a0a4d259799325b5e9"
format_stamp: "Formatted at 2026-02-02 15:04:13 on dist-test-slave-6zzp"
server_key: "9f9ce527f126acb08c7425bf672a0a74"
server_key_iv: "112ee10201070fae4f9cb5700ce8d9e2"
server_key_version: "encryptionkey@0"
I20260202 15:04:13.195755 8763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "ffb28890498b47a0a4d259799325b5e9"
format_stamp: "Formatted at 2026-02-02 15:04:13 on dist-test-slave-6zzp"
server_key: "9f9ce527f126acb08c7425bf672a0a74"
server_key_iv: "112ee10201070fae4f9cb5700ce8d9e2"
server_key_version: "encryptionkey@0"
I20260202 15:04:13.199182 8763 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260202 15:04:13.201347 8777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:13.202302 8763 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:13.202435 8763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "ffb28890498b47a0a4d259799325b5e9"
format_stamp: "Formatted at 2026-02-02 15:04:13 on dist-test-slave-6zzp"
server_key: "9f9ce527f126acb08c7425bf672a0a74"
server_key_iv: "112ee10201070fae4f9cb5700ce8d9e2"
server_key_version: "encryptionkey@0"
I20260202 15:04:13.202540 8763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:13.213268 8763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:13.213912 8763 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:13.214103 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:13.221468 8829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:38475 every 8 connection(s)
I20260202 15:04:13.221465 8763 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38475
I20260202 15:04:13.222546 8763 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:13.224041 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8763
I20260202 15:04:13.224192 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:13.224431 2705 external_mini_cluster.cc:1468] Setting key b5b6cf0ddb0c869aa65e0f954d00205e
I20260202 15:04:13.225654 8830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:13.232477 8830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9: Bootstrap starting.
I20260202 15:04:13.233106 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:13.225729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.2.164.126:38475)
0202 15:04:13.226385 (+ 656us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:13.226395 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:13.226437 (+ 42us) server_negotiation.cc:408] Connection header received
0202 15:04:13.227266 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:13.227289 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:13.227730 (+ 441us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:13.228094 (+ 364us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:13.228652 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:13.229864 (+ 1212us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:13.230558 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:13.230843 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:13.231258 (+ 415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:13.231313 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:13.231325 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:13.231798 (+ 473us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:13.231851 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:13.231869 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:13.231980 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:13.232089 (+ 109us) server_negotiation.cc:300] Negotiation successful
0202 15:04:13.232326 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260202 15:04:13.234994 8830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:13.235841 8830 log.cc:826] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:13.235965 2705 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260202 15:04:13.237962 8830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9: No bootstrap required, opened a new log
I20260202 15:04:13.240381 8830 raft_consensus.cc:359] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb28890498b47a0a4d259799325b5e9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38475 } }
I20260202 15:04:13.240592 8830 raft_consensus.cc:385] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:13.240674 8830 raft_consensus.cc:740] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb28890498b47a0a4d259799325b5e9, State: Initialized, Role: FOLLOWER
I20260202 15:04:13.241216 8830 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb28890498b47a0a4d259799325b5e9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38475 } }
I20260202 15:04:13.241379 8830 raft_consensus.cc:399] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:13.241468 8830 raft_consensus.cc:493] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:13.241592 8830 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:13.242420 8830 raft_consensus.cc:515] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb28890498b47a0a4d259799325b5e9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38475 } }
I20260202 15:04:13.242767 8830 leader_election.cc:304] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffb28890498b47a0a4d259799325b5e9; no voters:
I20260202 15:04:13.243067 8830 leader_election.cc:290] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:13.243203 8835 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:13.243451 8835 raft_consensus.cc:697] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [term 1 LEADER]: Becoming Leader. State: Replica: ffb28890498b47a0a4d259799325b5e9, State: Running, Role: LEADER
I20260202 15:04:13.243750 8835 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb28890498b47a0a4d259799325b5e9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38475 } }
I20260202 15:04:13.244233 8830 sys_catalog.cc:565] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:13.245292 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffb28890498b47a0a4d259799325b5e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb28890498b47a0a4d259799325b5e9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38475 } } }
I20260202 15:04:13.245422 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:13.245651 8837 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffb28890498b47a0a4d259799325b5e9. Latest consensus state: current_term: 1 leader_uuid: "ffb28890498b47a0a4d259799325b5e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb28890498b47a0a4d259799325b5e9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38475 } } }
I20260202 15:04:13.245748 8837 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:13.246109 8844 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:13.248726 8844 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:13.254317 8844 catalog_manager.cc:1357] Generated new cluster ID: 41082eb00ca04ce7b9c7da43a49d1b98
I20260202 15:04:13.254405 8844 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:13.277491 8844 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:13.278698 8844 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:13.283703 8844 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ffb28890498b47a0a4d259799325b5e9: Generated new TSK 0
I20260202 15:04:13.284404 8844 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:04:13.342888 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:13.297078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.2.164.126:38475)
0202 15:04:13.297218 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:13.297225 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:13.297382 (+ 157us) server_negotiation.cc:408] Connection header received
0202 15:04:13.297562 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:13.297564 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:13.297614 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:13.297694 (+ 80us) server_negotiation.cc:227] Negotiated authn=JWT
0202 15:04:13.298127 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:13.298630 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:13.299504 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:13.299644 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:13.299728 (+ 84us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0202 15:04:13.342418 (+ 42690us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0202 15:04:13.342738 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47814: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260202 15:04:13.342989 8833 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47814: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260202 15:04:13.343072 8859 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:13.296959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38475 (local address 127.0.0.1:47814)
0202 15:04:13.297295 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:13.297311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:13.297424 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:13.297730 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:13.297733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:13.297741 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0202 15:04:13.298017 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:13.298026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:13.298741 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:13.298745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:13.299400 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:13.299411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:13.299531 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:13.299553 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0202 15:04:13.342667 (+ 43114us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0202 15:04:13.342861 (+ 194us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:38475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":161,"threads_started":1}
I20260202 15:04:13.343976 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8763
2026-02-02T15:04:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (313 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:13 dist-test-slave-6zzp krb5kdc[8871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:13 dist-test-slave-6zzp krb5kdc[8871](info): set up 2 sockets
Feb 02 15:04:13 dist-test-slave-6zzp krb5kdc[8871](info): commencing operation
krb5kdc: starting...
W20260202 15:04:15.601714 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.990s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:15 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:15.750710 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44823
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:15.861634 8887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:15.861883 8887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:15.861939 8887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:15.865163 8887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:15.865221 8887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:15.865245 8887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:15.865264 8887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:15.865283 8887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:15.869663 8887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:15.870792 8887 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:15.871634 8887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:15.877588 8895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:15.877643 8893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:15.877718 8887 server_base.cc:1060] running on GCE node
W20260202 15:04:15.877600 8892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:15.878428 8887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:15.879364 8887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:15.880550 8887 hybrid_clock.cc:648] HybridClock initialized: now 1770044655880522 us; error 47 us; skew 500 ppm
Feb 02 15:04:15 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044655, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:15.884007 8887 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:04:15.885648 8887 webserver.cc:492] Webserver started at http://127.2.164.126:41393/ using document root <none> and password file <none>
I20260202 15:04:15.886340 8887 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:15.886418 8887 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:15.886632 8887 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:15.888437 8887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "c2589b66049349b0a9192aaf1114f6bd"
format_stamp: "Formatted at 2026-02-02 15:04:15 on dist-test-slave-6zzp"
server_key: "a4531681e910455bd2e374cbcc6c79ae"
server_key_iv: "9e81487e453ed7dc069ab51a056e5906"
server_key_version: "encryptionkey@0"
I20260202 15:04:15.888968 8887 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "c2589b66049349b0a9192aaf1114f6bd"
format_stamp: "Formatted at 2026-02-02 15:04:15 on dist-test-slave-6zzp"
server_key: "a4531681e910455bd2e374cbcc6c79ae"
server_key_iv: "9e81487e453ed7dc069ab51a056e5906"
server_key_version: "encryptionkey@0"
I20260202 15:04:15.892582 8887 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:15.895224 8902 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:15.896397 8887 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260202 15:04:15.896530 8887 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "c2589b66049349b0a9192aaf1114f6bd"
format_stamp: "Formatted at 2026-02-02 15:04:15 on dist-test-slave-6zzp"
server_key: "a4531681e910455bd2e374cbcc6c79ae"
server_key_iv: "9e81487e453ed7dc069ab51a056e5906"
server_key_version: "encryptionkey@0"
I20260202 15:04:15.896642 8887 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:15.927415 8887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:15.930470 8887 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:15.930689 8887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:15.938273 8887 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44823
I20260202 15:04:15.938288 8954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:44823 every 8 connection(s)
I20260202 15:04:15.939316 8887 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:15.942124 8955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:15.947225 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8887
I20260202 15:04:15.947388 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:15.947573 8955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd: Bootstrap starting.
I20260202 15:04:15.947619 2705 external_mini_cluster.cc:1468] Setting key 8e793cabc33a6f71f8c95ee1e6465384
I20260202 15:04:15.950116 8955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:15.950918 8955 log.cc:826] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:15.952605 8955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd: No bootstrap required, opened a new log
Feb 02 15:04:15 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:15.955431 8955 raft_consensus.cc:359] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2589b66049349b0a9192aaf1114f6bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44823 } }
I20260202 15:04:15.955628 8955 raft_consensus.cc:385] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:15.955708 8955 raft_consensus.cc:740] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2589b66049349b0a9192aaf1114f6bd, State: Initialized, Role: FOLLOWER
I20260202 15:04:15.956188 8955 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2589b66049349b0a9192aaf1114f6bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44823 } }
I20260202 15:04:15.956375 8955 raft_consensus.cc:399] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:15.956461 8955 raft_consensus.cc:493] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:15.956574 8955 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:15.957497 8955 raft_consensus.cc:515] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2589b66049349b0a9192aaf1114f6bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44823 } }
I20260202 15:04:15.957871 8955 leader_election.cc:304] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2589b66049349b0a9192aaf1114f6bd; no voters:
I20260202 15:04:15.958165 8955 leader_election.cc:290] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:15.958590 8960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:15.958815 8960 raft_consensus.cc:697] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [term 1 LEADER]: Becoming Leader. State: Replica: c2589b66049349b0a9192aaf1114f6bd, State: Running, Role: LEADER
I20260202 15:04:15.959149 8960 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2589b66049349b0a9192aaf1114f6bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44823 } }
I20260202 15:04:15.959394 8955 sys_catalog.cc:565] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:15.960659 8961 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2589b66049349b0a9192aaf1114f6bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2589b66049349b0a9192aaf1114f6bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44823 } } }
I20260202 15:04:15.960779 8961 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:15.960922 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:15.948835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.2.164.126:44823)
0202 15:04:15.949326 (+ 491us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:15.949338 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:15.949377 (+ 39us) server_negotiation.cc:408] Connection header received
0202 15:04:15.950073 (+ 696us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:15.950099 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:15.950463 (+ 364us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:15.950800 (+ 337us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:15.951611 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:15.952339 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:15.953118 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:15.953412 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:15.955648 (+ 2236us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:15.955669 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:15.955683 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:15.955760 (+ 77us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:15.957761 (+ 2001us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:15.958269 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:15.958275 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:15.958280 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:15.958379 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:15.958659 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:15.958662 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:15.958664 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:15.959089 (+ 425us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:15.959273 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:15.959839 (+ 566us) server_negotiation.cc:300] Negotiation successful
0202 15:04:15.960107 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":180,"threads_started":1}
I20260202 15:04:15.961310 8962 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2589b66049349b0a9192aaf1114f6bd. Latest consensus state: current_term: 1 leader_uuid: "c2589b66049349b0a9192aaf1114f6bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2589b66049349b0a9192aaf1114f6bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44823 } } }
I20260202 15:04:15.961452 8962 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd [sys.catalog]: This master's current role is: LEADER
W20260202 15:04:15.964269 8975 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:04:15.964346 8975 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:04:15.964422 8969 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:15.965226 8969 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:15.974589 8969 catalog_manager.cc:1357] Generated new cluster ID: b5a8c68feac748a5b61ff3f3d8e7436d
I20260202 15:04:15.974685 8969 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:15.981127 8969 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:15.982031 8969 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:15.991080 8969 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c2589b66049349b0a9192aaf1114f6bd: Generated new TSK 0
I20260202 15:04:15.991777 8969 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:16.046499 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44823
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:16.155359 8983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:16.155604 8983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:16.155666 8983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:16.159068 8983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:16.159142 8983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:16.159291 8983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:16.163956 8983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:44823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:16.165190 8983 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:16.166112 8983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:16.173324 8988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:16.173324 8989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:16.173348 8991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:16.174078 8983 server_base.cc:1060] running on GCE node
I20260202 15:04:16.174595 8983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:16.175274 8983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:16.176434 8983 hybrid_clock.cc:648] HybridClock initialized: now 1770044656176423 us; error 34 us; skew 500 ppm
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.179418 8983 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:04:16.180536 8983 webserver.cc:492] Webserver started at http://127.2.164.65:35165/ using document root <none> and password file <none>
I20260202 15:04:16.181205 8983 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:16.181257 8983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:16.181461 8983 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:16.183259 8983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "50d537409e5a454f860562b8b7f7d74e"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "ade3bfc15de1cf9ee2365403222243f0"
server_key_iv: "c732d1a3bc0018390610718a2d48c60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.183763 8983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "50d537409e5a454f860562b8b7f7d74e"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "ade3bfc15de1cf9ee2365403222243f0"
server_key_iv: "c732d1a3bc0018390610718a2d48c60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.187268 8983 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260202 15:04:16.189482 8998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.190449 8983 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:16.190583 8983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "50d537409e5a454f860562b8b7f7d74e"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "ade3bfc15de1cf9ee2365403222243f0"
server_key_iv: "c732d1a3bc0018390610718a2d48c60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.190686 8983 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:16.203199 8983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:16.206049 8983 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:16.206254 8983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:16.206832 8983 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:16.207723 8983 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:16.207804 8983 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.207878 8983 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:16.207923 8983 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.217206 8983 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:46331
I20260202 15:04:16.217254 9111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:46331 every 8 connection(s)
I20260202 15:04:16.218165 8983 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:16.222797 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 8983
I20260202 15:04:16.222959 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:16.223274 2705 external_mini_cluster.cc:1468] Setting key 87c995eb77cbe5b4c81c7e29080869da
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:16.231693 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.219891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34813 (local address 127.2.164.126:44823)
0202 15:04:16.220060 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.220065 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.220790 (+ 725us) server_negotiation.cc:408] Connection header received
0202 15:04:16.221623 (+ 833us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.221626 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.221676 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.221752 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.223730 (+ 1978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.224340 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.225148 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.225371 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.227637 (+ 2266us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:16.227662 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:16.227668 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:16.227711 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:16.229473 (+ 1762us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.230147 (+ 674us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.230150 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.230151 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.230198 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.230534 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.230536 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.230537 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.230699 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:16.230778 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:16.231441 (+ 663us) server_negotiation.cc:300] Negotiation successful
0202 15:04:16.231561 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:04:16.232662 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.220213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44823 (local address 127.2.164.65:34813)
0202 15:04:16.220637 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:16.220677 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:16.221441 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:16.221894 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:16.221902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:16.222282 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:16.223523 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:16.223540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.224515 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.224519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:16.224981 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:16.224990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.225224 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.225826 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:16.225851 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:16.227479 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:16.229605 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:16.229614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:16.229631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:16.230019 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:16.230302 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:16.230307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:16.230310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:16.230440 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:16.230809 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:16.230819 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:16.231162 (+ 343us) client_negotiation.cc:769] Sending connection context
0202 15:04:16.231423 (+ 261us) client_negotiation.cc:240] Negotiation successful
0202 15:04:16.231750 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260202 15:04:16.234081 9112 heartbeater.cc:344] Connected to a master server at 127.2.164.126:44823
I20260202 15:04:16.234324 9112 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:16.235047 9112 heartbeater.cc:507] Master 127.2.164.126:44823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:04:16.236667 8919 ts_manager.cc:194] Registered new tserver with Master: 50d537409e5a454f860562b8b7f7d74e (127.2.164.65:46331)
I20260202 15:04:16.237957 8919 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:34813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:16.276865 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44823
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:16.385535 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:16.385758 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:16.385818 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:16.389309 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:16.389410 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:16.389515 9119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:16.393968 9119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:44823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:16.395200 9119 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:16.396023 9119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:16.402524 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:16.402524 9125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:16.402551 9127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:16.403321 9119 server_base.cc:1060] running on GCE node
I20260202 15:04:16.403676 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:16.404181 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:16.405340 9119 hybrid_clock.cc:648] HybridClock initialized: now 1770044656405305 us; error 51 us; skew 500 ppm
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.407835 9119 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:04:16.408867 9119 webserver.cc:492] Webserver started at http://127.2.164.66:36853/ using document root <none> and password file <none>
I20260202 15:04:16.409442 9119 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:16.409493 9119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:16.409644 9119 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:16.411274 9119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "651ef7bc4c83488fb62f7a8d7db9d4d6"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "8dc2c4f906a9d6a21f0ef396bcb27608"
server_key_iv: "254582aeac0d6c5a84a656e8a099b901"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.411702 9119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "651ef7bc4c83488fb62f7a8d7db9d4d6"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "8dc2c4f906a9d6a21f0ef396bcb27608"
server_key_iv: "254582aeac0d6c5a84a656e8a099b901"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.415062 9119 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:04:16.417174 9134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.418218 9119 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:16.418347 9119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "651ef7bc4c83488fb62f7a8d7db9d4d6"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "8dc2c4f906a9d6a21f0ef396bcb27608"
server_key_iv: "254582aeac0d6c5a84a656e8a099b901"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.418447 9119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:16.437929 9119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:16.441129 9119 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:16.441329 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:16.441907 9119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:16.442884 9119 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:16.442957 9119 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.443038 9119 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:16.443079 9119 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.452566 9119 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:36587
I20260202 15:04:16.452597 9247 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:36587 every 8 connection(s)
I20260202 15:04:16.453619 9119 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:16.463227 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9119
I20260202 15:04:16.463348 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:16.463639 2705 external_mini_cluster.cc:1468] Setting key a7e8eed32c83fc883524d9bc96985c22
I20260202 15:04:16.466943 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.455488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45121 (local address 127.2.164.126:44823)
0202 15:04:16.455639 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.455643 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.456326 (+ 683us) server_negotiation.cc:408] Connection header received
0202 15:04:16.457246 (+ 920us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.457250 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.457312 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.457416 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.459072 (+ 1656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.459579 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.460264 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.460446 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.463208 (+ 2762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:16.463235 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:16.463257 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:16.463291 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:16.464848 (+ 1557us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.465461 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.465465 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.465467 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.465519 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.465840 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.465844 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.465846 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.466015 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:16.466121 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:16.466635 (+ 514us) server_negotiation.cc:300] Negotiation successful
0202 15:04:16.466763 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260202 15:04:16.467608 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.455754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44823 (local address 127.2.164.66:45121)
0202 15:04:16.456190 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:16.456228 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:16.456993 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:16.457583 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:16.457591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:16.458001 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:16.458919 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:16.458933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.459724 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.459727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:16.460142 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:16.460149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.460643 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.461386 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:16.461414 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:16.463015 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:16.465062 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:16.465067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:16.465079 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:16.465321 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:16.465620 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:16.465623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:16.465625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:16.465739 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:16.466122 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:16.466130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:16.466385 (+ 255us) client_negotiation.cc:769] Sending connection context
0202 15:04:16.466590 (+ 205us) client_negotiation.cc:240] Negotiation successful
0202 15:04:16.466881 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260202 15:04:16.468813 9248 heartbeater.cc:344] Connected to a master server at 127.2.164.126:44823
I20260202 15:04:16.469129 9248 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:16.469828 9248 heartbeater.cc:507] Master 127.2.164.126:44823 requested a full tablet report, sending...
I20260202 15:04:16.471082 8919 ts_manager.cc:194] Registered new tserver with Master: 651ef7bc4c83488fb62f7a8d7db9d4d6 (127.2.164.66:36587)
I20260202 15:04:16.471892 8919 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:45121
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:16.516587 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44823
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:16.632066 9255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:16.632294 9255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:16.632355 9255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:16.635725 9255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:16.635802 9255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:16.635887 9255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:16.640393 9255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:44823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:16.641646 9255 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:16.642673 9255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:16.649672 9263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:16.649708 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:16.649921 9255 server_base.cc:1060] running on GCE node
W20260202 15:04:16.649685 9261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:16.650497 9255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:16.651098 9255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:16.652290 9255 hybrid_clock.cc:648] HybridClock initialized: now 1770044656652278 us; error 25 us; skew 500 ppm
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.655073 9255 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:04:16.656222 9255 webserver.cc:492] Webserver started at http://127.2.164.67:37139/ using document root <none> and password file <none>
I20260202 15:04:16.656817 9255 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:16.656895 9255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:16.657171 9255 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:16.658964 9255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "2a8a5994e41640df8fabb88ea2e29230"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "b2f2174c7bd93edb409966e7394a05d8"
server_key_iv: "4848d89f9832083aefd9fa1520fe656f"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.659468 9255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "2a8a5994e41640df8fabb88ea2e29230"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "b2f2174c7bd93edb409966e7394a05d8"
server_key_iv: "4848d89f9832083aefd9fa1520fe656f"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.663403 9255 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260202 15:04:16.665827 9270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.666952 9255 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:16.667097 9255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "2a8a5994e41640df8fabb88ea2e29230"
format_stamp: "Formatted at 2026-02-02 15:04:16 on dist-test-slave-6zzp"
server_key: "b2f2174c7bd93edb409966e7394a05d8"
server_key_iv: "4848d89f9832083aefd9fa1520fe656f"
server_key_version: "encryptionkey@0"
I20260202 15:04:16.667208 9255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:16.709075 9255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:16.715776 9255 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:16.716089 9255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:16.716982 9255 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:16.718402 9255 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:16.718466 9255 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.718552 9255 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:16.718587 9255 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:16.728631 9255 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34053
I20260202 15:04:16.728662 9383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:34053 every 8 connection(s)
I20260202 15:04:16.729730 9255 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:16.733233 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9255
I20260202 15:04:16.733402 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:16.733723 2705 external_mini_cluster.cc:1468] Setting key 98d83d6651f314f16ab34ccd13602ff2
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:16.742681 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.731615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:51153 (local address 127.2.164.126:44823)
0202 15:04:16.731796 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.731800 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.732562 (+ 762us) server_negotiation.cc:408] Connection header received
0202 15:04:16.733684 (+ 1122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.733688 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.733763 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.733867 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.735637 (+ 1770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.736162 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.736831 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.737039 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.739277 (+ 2238us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:16.739303 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:16.739305 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:16.739333 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:16.740596 (+ 1263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.741166 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.741170 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.741174 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.741225 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.741533 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.741536 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.741538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.741708 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:16.741827 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:16.742365 (+ 538us) server_negotiation.cc:300] Negotiation successful
0202 15:04:16.742498 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260202 15:04:16.743427 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.731870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44823 (local address 127.2.164.67:51153)
0202 15:04:16.732411 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:16.732450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:16.733430 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:16.734109 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:16.734119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:16.734525 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:16.735481 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:16.735496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.736296 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.736306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:16.736709 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:16.736715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.736969 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.737600 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:16.737628 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:16.739122 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:16.740731 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:16.740737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:16.740750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:16.741067 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:16.741325 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:16.741328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:16.741331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:16.741433 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:16.741838 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:16.741846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:16.742106 (+ 260us) client_negotiation.cc:769] Sending connection context
0202 15:04:16.742317 (+ 211us) client_negotiation.cc:240] Negotiation successful
0202 15:04:16.742611 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":112,"threads_started":1}
I20260202 15:04:16.744582 9384 heartbeater.cc:344] Connected to a master server at 127.2.164.126:44823
I20260202 15:04:16.744845 9384 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:16.745442 9384 heartbeater.cc:507] Master 127.2.164.126:44823 requested a full tablet report, sending...
I20260202 15:04:16.746516 8919 ts_manager.cc:194] Registered new tserver with Master: 2a8a5994e41640df8fabb88ea2e29230 (127.2.164.67:34053)
I20260202 15:04:16.747159 8919 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:51153
I20260202 15:04:16.747840 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/test-admin.keytab.
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.765280 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260202 15:04:16.767944 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:16.766658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.2.164.65:46331)
0202 15:04:16.766919 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.766925 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.766949 (+ 24us) server_negotiation.cc:408] Connection header received
0202 15:04:16.767014 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.767019 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.767186 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.767306 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.767735 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59552: BlockingRecv error: recv got EOF from 127.0.0.1:59552 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
W20260202 15:04:16.768172 2705 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc)
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.769331 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.770314 2705 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:04:16.779276 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.771308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.2.164.65:46331)
0202 15:04:16.771427 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.771431 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.771485 (+ 54us) server_negotiation.cc:408] Connection header received
0202 15:04:16.771587 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.771590 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.771634 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.771706 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.772471 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.773249 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.773865 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.774044 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.776152 (+ 2108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:16.776193 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:16.776206 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:16.776231 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:16.777916 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.778277 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.778282 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.778283 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.778334 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.778519 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.778521 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.778522 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.778695 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:16.778849 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:16.779019 (+ 170us) server_negotiation.cc:300] Negotiation successful
0202 15:04:16.779139 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260202 15:04:16.782954 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:16.781991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.2.164.65:46331)
0202 15:04:16.782125 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.782129 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.782223 (+ 94us) server_negotiation.cc:408] Connection header received
0202 15:04:16.782339 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.782342 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.782387 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.782471 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.782844 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59580: BlockingRecv error: recv got EOF from 127.0.0.1:59580 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260202 15:04:16.783113 2705 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc' of type FILE
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.784137 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.785051 2705 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:04:16.792872 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.785791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.2.164.65:46331)
0202 15:04:16.785914 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.785919 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.786039 (+ 120us) server_negotiation.cc:408] Connection header received
0202 15:04:16.786120 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.786122 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.786158 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.786221 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.786766 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.787231 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.787778 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.787896 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.790263 (+ 2367us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:16.790280 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:16.790282 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:16.790305 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:16.791516 (+ 1211us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.791867 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.791870 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.791871 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.791909 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.792150 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.792152 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.792153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.792277 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:16.792373 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:16.792560 (+ 187us) server_negotiation.cc:300] Negotiation successful
0202 15:04:16.792680 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260202 15:04:16.795691 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:16.794630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.2.164.65:46331)
0202 15:04:16.794770 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.794774 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.794863 (+ 89us) server_negotiation.cc:408] Connection header received
0202 15:04:16.795025 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.795028 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.795069 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.795159 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.795566 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59590: BlockingRecv error: recv got EOF from 127.0.0.1:59590 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260202 15:04:16.795832 2705 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770044596670313-2705-0/krb5kdc/krb5cc' of type FILE
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.796836 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:16.797727 2705 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[8871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260202 15:04:16.805220 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:16.798480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.2.164.65:46331)
0202 15:04:16.798631 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:16.798635 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:16.798710 (+ 75us) server_negotiation.cc:408] Connection header received
0202 15:04:16.798791 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:16.798793 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:16.798829 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:16.798893 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:16.799431 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.799878 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:16.800456 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:16.800573 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:16.802570 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:16.802584 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:16.802586 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:16.802609 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:16.803795 (+ 1186us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.804190 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.804193 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.804194 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.804231 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:16.804467 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:16.804469 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:16.804470 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:16.804585 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:16.804691 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:16.804868 (+ 177us) server_negotiation.cc:300] Negotiation successful
0202 15:04:16.804991 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260202 15:04:16.806425 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8983
I20260202 15:04:16.812208 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9119
I20260202 15:04:16.817863 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9255
I20260202 15:04:16.823180 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 8887
2026-02-02T15:04:16Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3235 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[9404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[9404](info): set up 2 sockets
Feb 02 15:04:16 dist-test-slave-6zzp krb5kdc[9404](info): commencing operation
krb5kdc: starting...
W20260202 15:04:18.846014 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.999s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:18 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:19.010661 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:45883
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:19.126111 9420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:19.126405 9420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:19.126494 9420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:19.129877 9420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:19.129994 9420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:19.130036 9420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:19.130081 9420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:19.130129 9420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:19.134658 9420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:45883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:19.135914 9420 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:19.136849 9420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:19.142895 9425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.142839 9428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.142889 9426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:19.143347 9420 server_base.cc:1060] running on GCE node
I20260202 15:04:19.143850 9420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:19.144762 9420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:19.145944 9420 hybrid_clock.cc:648] HybridClock initialized: now 1770044659145925 us; error 45 us; skew 500 ppm
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:19.148535 9420 init.cc:377] Logged in from keytab as oryx/127.2.164.126@KRBTEST.COM (short username oryx)
I20260202 15:04:19.149883 9420 webserver.cc:492] Webserver started at http://127.2.164.126:34281/ using document root <none> and password file <none>
I20260202 15:04:19.150466 9420 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:19.150544 9420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:19.150763 9420 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:19.152570 9420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "94296279378342b0afd24fa731eebc9f"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "098f7508489ee4cf00d978e2c3d747dd"
server_key_iv: "5027542ae68b0ad77a568eefe9dd3104"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.153169 9420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "94296279378342b0afd24fa731eebc9f"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "098f7508489ee4cf00d978e2c3d747dd"
server_key_iv: "5027542ae68b0ad77a568eefe9dd3104"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.156708 9420 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:04:19.159090 9435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.160241 9420 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:19.160398 9420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "94296279378342b0afd24fa731eebc9f"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "098f7508489ee4cf00d978e2c3d747dd"
server_key_iv: "5027542ae68b0ad77a568eefe9dd3104"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.160513 9420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:19.185745 9420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:19.188937 9420 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:19.189218 9420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:19.197283 9420 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:45883
I20260202 15:04:19.197279 9487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:45883 every 8 connection(s)
I20260202 15:04:19.198349 9420 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:19.201092 9488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:19.206629 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Bootstrap starting.
I20260202 15:04:19.207500 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9420
I20260202 15:04:19.207657 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:19.207922 2705 external_mini_cluster.cc:1468] Setting key 23a55f2262b4cee52af352c8e9fd6df7
I20260202 15:04:19.209367 9488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:19.210088 9488 log.cc:826] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:19.212288 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: No bootstrap required, opened a new log
I20260202 15:04:19.215162 9488 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } }
I20260202 15:04:19.215389 9488 raft_consensus.cc:385] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:19.215476 9488 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94296279378342b0afd24fa731eebc9f, State: Initialized, Role: FOLLOWER
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:19.216082 9488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } }
I20260202 15:04:19.216220 9488 raft_consensus.cc:399] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:19.216334 9488 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:19.216456 9488 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:19.217438 9488 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } }
I20260202 15:04:19.217840 9488 leader_election.cc:304] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94296279378342b0afd24fa731eebc9f; no voters:
I20260202 15:04:19.218170 9488 leader_election.cc:290] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:19.218577 9493 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:19.218786 9493 raft_consensus.cc:697] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 LEADER]: Becoming Leader. State: Replica: 94296279378342b0afd24fa731eebc9f, State: Running, Role: LEADER
I20260202 15:04:19.219182 9493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } }
I20260202 15:04:19.219414 9488 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:19.220930 9494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } }
I20260202 15:04:19.220932 9495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94296279378342b0afd24fa731eebc9f. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } }
I20260202 15:04:19.221086 9494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:19.221104 9495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:19.222268 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.209106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.2.164.126:45883)
0202 15:04:19.209568 (+ 462us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:19.209578 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:19.209613 (+ 35us) server_negotiation.cc:408] Connection header received
0202 15:04:19.210176 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:19.210193 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:19.210531 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:19.210837 (+ 306us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:19.211668 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.212883 (+ 1215us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.213641 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.213921 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.216366 (+ 2445us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:19.216407 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:19.216425 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:19.216477 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:19.218627 (+ 2150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.219164 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.219170 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.219176 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.219277 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.219547 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.219550 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.219552 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.219979 (+ 427us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:19.220952 (+ 973us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:19.221153 (+ 201us) server_negotiation.cc:300] Negotiation successful
0202 15:04:19.221437 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":172,"threads_started":1}
W20260202 15:04:19.224304 9508 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:04:19.224406 9508 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:04:19.224509 9504 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:19.225435 9504 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:19.230695 9504 catalog_manager.cc:1357] Generated new cluster ID: dbfff5e6ac374cf4b4b93b06201d8cd4
I20260202 15:04:19.230774 9504 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:19.241950 9504 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:19.242996 9504 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:19.257177 9504 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Generated new TSK 0
I20260202 15:04:19.258150 9504 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:19.318310 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:45883
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:19.433197 9516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:19.433537 9516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:19.433724 9516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:19.437712 9516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:19.437795 9516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:19.437880 9516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:19.442534 9516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:45883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:19.443580 9516 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:19.444492 9516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:19.451069 9524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.451071 9521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.451184 9522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:19.451363 9516 server_base.cc:1060] running on GCE node
I20260202 15:04:19.451820 9516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:19.452378 9516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:19.453534 9516 hybrid_clock.cc:648] HybridClock initialized: now 1770044659453509 us; error 42 us; skew 500 ppm
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:19.456347 9516 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260202 15:04:19.457545 9516 webserver.cc:492] Webserver started at http://127.2.164.65:38255/ using document root <none> and password file <none>
I20260202 15:04:19.458163 9516 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:19.458237 9516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:19.458442 9516 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:19.460151 9516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "604b79fe893d4460ad4c782af7e87396"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "29fd5dac4d168ba37d06259bc4671f3e"
server_key_iv: "2726f4f14dcf920fdfab0ad89406ef35"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.460634 9516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "604b79fe893d4460ad4c782af7e87396"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "29fd5dac4d168ba37d06259bc4671f3e"
server_key_iv: "2726f4f14dcf920fdfab0ad89406ef35"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.464046 9516 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260202 15:04:19.466363 9531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.467450 9516 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:04:19.467595 9516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "604b79fe893d4460ad4c782af7e87396"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "29fd5dac4d168ba37d06259bc4671f3e"
server_key_iv: "2726f4f14dcf920fdfab0ad89406ef35"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.467705 9516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:19.482534 9516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:19.485767 9516 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:19.485989 9516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:19.486586 9516 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:19.487519 9516 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:19.487591 9516 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.487658 9516 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:19.487707 9516 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.497237 9516 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:38971
I20260202 15:04:19.497278 9644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:38971 every 8 connection(s)
I20260202 15:04:19.498255 9516 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:19.505105 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9516
I20260202 15:04:19.505236 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:19.505494 2705 external_mini_cluster.cc:1468] Setting key 03d77786673ca189572c0fb1ee4d3514
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:19.511653 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.500107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:46021 (local address 127.2.164.126:45883)
0202 15:04:19.500290 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:19.500295 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:19.501153 (+ 858us) server_negotiation.cc:408] Connection header received
0202 15:04:19.501998 (+ 845us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:19.502002 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:19.502051 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:19.502132 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:19.503845 (+ 1713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.504362 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.505091 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.505299 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.507821 (+ 2522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:19.507850 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:19.507853 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:19.507877 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:19.509471 (+ 1594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.510094 (+ 623us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.510097 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.510099 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.510145 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.510503 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.510505 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.510506 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.510653 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:19.510727 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:19.511416 (+ 689us) server_negotiation.cc:300] Negotiation successful
0202 15:04:19.511530 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260202 15:04:19.512696 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.500381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:46021)
0202 15:04:19.500959 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:19.501048 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:19.501796 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:19.502287 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:19.502296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:19.502701 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:19.503680 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:19.503695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.504477 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.504480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:19.504914 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:19.504922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.505232 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.505993 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:19.506018 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:19.507668 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:19.509596 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:19.509604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:19.509619 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:19.509978 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:19.510258 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:19.510264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:19.510269 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:19.510408 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:19.510765 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:19.510775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:19.511129 (+ 354us) client_negotiation.cc:769] Sending connection context
0202 15:04:19.511393 (+ 264us) client_negotiation.cc:240] Negotiation successful
0202 15:04:19.511704 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":123,"threads_started":1}
I20260202 15:04:19.513958 9645 heartbeater.cc:344] Connected to a master server at 127.2.164.126:45883
I20260202 15:04:19.514241 9645 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:19.514945 9645 heartbeater.cc:507] Master 127.2.164.126:45883 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:04:19.516559 9449 ts_manager.cc:194] Registered new tserver with Master: 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971)
I20260202 15:04:19.517988 9449 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:46021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:19.561118 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:45883
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:19.667876 9652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:19.668165 9652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:19.668258 9652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:19.671515 9652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:19.671620 9652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:19.671759 9652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:19.676017 9652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:45883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:19.677130 9652 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:19.678013 9652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:19.684638 9660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.684638 9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.684662 9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:19.685621 9652 server_base.cc:1060] running on GCE node
I20260202 15:04:19.686365 9652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:19.686941 9652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:19.688090 9652 hybrid_clock.cc:648] HybridClock initialized: now 1770044659688066 us; error 42 us; skew 500 ppm
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:19.690713 9652 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260202 15:04:19.691829 9652 webserver.cc:492] Webserver started at http://127.2.164.66:42519/ using document root <none> and password file <none>
I20260202 15:04:19.692394 9652 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:19.692469 9652 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:19.692680 9652 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:19.694473 9652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "aca6cb50f08848fcad0733a5a39d4282"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "cae981eec9a365e45548bf11a96d57f9"
server_key_iv: "283e035dd248e910fb1f9ac15b64c15b"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.694962 9652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "aca6cb50f08848fcad0733a5a39d4282"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "cae981eec9a365e45548bf11a96d57f9"
server_key_iv: "283e035dd248e910fb1f9ac15b64c15b"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.698434 9652 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:19.700745 9667 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.701896 9652 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:19.702033 9652 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "aca6cb50f08848fcad0733a5a39d4282"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "cae981eec9a365e45548bf11a96d57f9"
server_key_iv: "283e035dd248e910fb1f9ac15b64c15b"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.702162 9652 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:19.719611 9652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:19.722970 9652 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:19.723191 9652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:19.723749 9652 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:19.724658 9652 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:19.724707 9652 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.724777 9652 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:19.724813 9652 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.733858 9652 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:32967
I20260202 15:04:19.733886 9780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:32967 every 8 connection(s)
I20260202 15:04:19.734838 9652 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:19.737293 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9652
I20260202 15:04:19.737408 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:19.737671 2705 external_mini_cluster.cc:1468] Setting key e0c3abc4e3894fce7f62953b83477dd3
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:04:19.747965 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.736630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:51071 (local address 127.2.164.126:45883)
0202 15:04:19.736786 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:19.736790 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:19.737593 (+ 803us) server_negotiation.cc:408] Connection header received
0202 15:04:19.738616 (+ 1023us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:19.738619 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:19.738675 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:19.738760 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:19.740630 (+ 1870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.741200 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.741878 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.742066 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.744307 (+ 2241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:19.744332 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:19.744335 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:19.744362 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:19.745856 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.746353 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.746357 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.746359 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.746411 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.746710 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.746714 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.746716 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.746883 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:19.746991 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:19.747639 (+ 648us) server_negotiation.cc:300] Negotiation successful
0202 15:04:19.747779 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:04:19.748858 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.736905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:51071)
0202 15:04:19.737435 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:19.737486 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:19.738416 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:19.738950 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:19.738961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:19.739402 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:19.740447 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:19.740468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.741316 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.741320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:19.741741 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:19.741749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.742002 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.742658 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:19.742692 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:19.744157 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:19.745980 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:19.745986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:19.745999 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:19.746246 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:19.746506 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:19.746509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:19.746511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:19.746615 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:19.746991 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:19.747000 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:19.747344 (+ 344us) client_negotiation.cc:769] Sending connection context
0202 15:04:19.747607 (+ 263us) client_negotiation.cc:240] Negotiation successful
0202 15:04:19.747907 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.67@KRBTEST.COM" created.
I20260202 15:04:19.750247 9781 heartbeater.cc:344] Connected to a master server at 127.2.164.126:45883
I20260202 15:04:19.750526 9781 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:19.751204 9781 heartbeater.cc:507] Master 127.2.164.126:45883 requested a full tablet report, sending...
I20260202 15:04:19.752444 9449 ts_manager.cc:194] Registered new tserver with Master: aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967)
I20260202 15:04:19.753336 9449 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:51071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:19.788297 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:45883
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:19.896605 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:19.896890 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:19.896993 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:19.900326 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:19.900424 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:19.900547 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:19.904812 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:45883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:19.905882 9788 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:19.906776 9788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:19.913501 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.913539 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:19.913542 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:19.913683 9788 server_base.cc:1060] running on GCE node
I20260202 15:04:19.914100 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:19.914932 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:19.916117 9788 hybrid_clock.cc:648] HybridClock initialized: now 1770044659916094 us; error 44 us; skew 500 ppm
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:19.919170 9788 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260202 15:04:19.920239 9788 webserver.cc:492] Webserver started at http://127.2.164.67:38555/ using document root <none> and password file <none>
I20260202 15:04:19.920814 9788 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:19.920888 9788 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:19.921197 9788 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:19.923416 9788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "a30432df9d09541a9eee2a16cbef2c0f"
server_key_iv: "68dccfe2c7bc52902a67bc6a66c77573"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.923916 9788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "a30432df9d09541a9eee2a16cbef2c0f"
server_key_iv: "68dccfe2c7bc52902a67bc6a66c77573"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.927765 9788 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260202 15:04:19.930042 9803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.931001 9788 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:19.931141 9788 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "a30432df9d09541a9eee2a16cbef2c0f"
server_key_iv: "68dccfe2c7bc52902a67bc6a66c77573"
server_key_version: "encryptionkey@0"
I20260202 15:04:19.931243 9788 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:19.944473 9788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:19.948781 9788 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:19.949081 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:19.949671 9788 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:19.950757 9788 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:19.950837 9788 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.950935 9788 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:19.950994 9788 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:19.960705 9788 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45485
I20260202 15:04:19.960719 9916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:45485 every 8 connection(s)
I20260202 15:04:19.961800 9788 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:19.964027 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9788
I20260202 15:04:19.964148 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:19.964573 2705 external_mini_cluster.cc:1468] Setting key 892e18f5b7237e30b4c4003ce1c50625
Feb 02 15:04:19 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:19.975329 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.963685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:51333 (local address 127.2.164.126:45883)
0202 15:04:19.963860 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:19.963865 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:19.964724 (+ 859us) server_negotiation.cc:408] Connection header received
0202 15:04:19.965736 (+ 1012us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:19.965740 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:19.965800 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:19.965892 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:19.967831 (+ 1939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.968343 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.969364 (+ 1021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.969508 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.971820 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:19.971839 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:19.971842 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:19.971870 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:19.973478 (+ 1608us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.973963 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.973966 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.973968 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.974010 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:19.974328 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:19.974330 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:19.974331 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:19.974481 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:19.974554 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:19.975097 (+ 543us) server_negotiation.cc:300] Negotiation successful
0202 15:04:19.975199 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260202 15:04:19.976013 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:19.964017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:51333)
0202 15:04:19.964538 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:19.964582 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:19.965499 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:19.966081 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:19.966089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:19.966649 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:19.967682 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:19.967696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.968592 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:19.968598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:19.969223 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:19.969237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:19.969428 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:19.970001 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:19.970030 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:19.971648 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:19.973600 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:19.973605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:19.973615 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:19.973869 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:19.974130 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:19.974133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:19.974134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:19.974244 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:19.974571 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:19.974577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:19.974851 (+ 274us) client_negotiation.cc:769] Sending connection context
0202 15:04:19.975033 (+ 182us) client_negotiation.cc:240] Negotiation successful
0202 15:04:19.975256 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":152,"threads_started":1}
I20260202 15:04:19.977245 9917 heartbeater.cc:344] Connected to a master server at 127.2.164.126:45883
I20260202 15:04:19.977552 9917 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:19.978274 9917 heartbeater.cc:507] Master 127.2.164.126:45883 requested a full tablet report, sending...
I20260202 15:04:19.979370 9449 ts_manager.cc:194] Registered new tserver with Master: 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485)
I20260202 15:04:19.979885 9449 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:51333
I20260202 15:04:19.990466 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.164.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:20.041523 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:37549
--webserver_interface=127.2.164.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:45883,127.2.164.125:37549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:20.148968 9924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:20.149296 9924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:20.149403 9924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:20.152662 9924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:20.152750 9924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:20.152774 9924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:20.152824 9924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:20.152863 9924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:20.157311 9924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:45883,127.2.164.125:37549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.164.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
W20260202 15:04:20.157542 9924 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.164.126:45883,127.2.164.125:37549'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260202 15:04:20.158475 9924 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:20.159334 9924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:20.165064 9932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:20.165093 9929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:20.165210 9924 server_base.cc:1060] running on GCE node
W20260202 15:04:20.165355 9930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:20.165841 9924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:20.166841 9924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:20.168016 9924 hybrid_clock.cc:648] HybridClock initialized: now 1770044660167985 us; error 47 us; skew 500 ppm
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:20.170778 9924 init.cc:377] Logged in from keytab as oryx/127.2.164.125@KRBTEST.COM (short username oryx)
I20260202 15:04:20.171932 9924 webserver.cc:492] Webserver started at http://127.2.164.125:40641/ using document root <none> and password file <none>
I20260202 15:04:20.172477 9924 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:20.172525 9924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:20.172685 9924 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:20.174453 9924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/instance:
uuid: "f11e9461627f442096e6fdcba2f270b7"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "cf7c6c596c83814d4806eeb1aa744b66"
server_key_iv: "dd6ab20375c5036bdc7f2dc0baed8cf1"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.174891 9924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal/instance:
uuid: "f11e9461627f442096e6fdcba2f270b7"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "cf7c6c596c83814d4806eeb1aa744b66"
server_key_iv: "dd6ab20375c5036bdc7f2dc0baed8cf1"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.178565 9924 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260202 15:04:20.180713 9939 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:20.181835 9924 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:20.181967 9924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
uuid: "f11e9461627f442096e6fdcba2f270b7"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "cf7c6c596c83814d4806eeb1aa744b66"
server_key_iv: "dd6ab20375c5036bdc7f2dc0baed8cf1"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.182070 9924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:20.195222 9924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:20.198832 9924 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:20.199023 9924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:20.207588 9924 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:37549
I20260202 15:04:20.207617 9991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.125:37549 every 8 connection(s)
I20260202 15:04:20.208789 9924 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/info.pb
I20260202 15:04:20.212011 9992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:20.216346 9992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
I20260202 15:04:20.217523 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 9924
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260202 15:04:20.228904 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.218292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.2.164.126:45883)
0202 15:04:20.218449 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.218454 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.219391 (+ 937us) server_negotiation.cc:408] Connection header received
0202 15:04:20.220112 (+ 721us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.220115 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.220178 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.220281 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.221269 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.222013 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.222675 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.222842 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.225207 (+ 2365us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.225233 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.225236 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.225262 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.226874 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.227336 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.227343 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.227346 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.227400 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.227700 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.227703 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.227704 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.227877 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.227997 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.228587 (+ 590us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.228715 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260202 15:04:20.229619 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.218589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.2.164.125:37549)
0202 15:04:20.219095 (+ 506us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.219105 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.219139 (+ 34us) server_negotiation.cc:408] Connection header received
0202 15:04:20.219814 (+ 675us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.219837 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.220018 (+ 181us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.220230 (+ 212us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.221225 (+ 995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.222003 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.222638 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.222894 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.225613 (+ 2719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.225631 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.225660 (+ 29us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.225688 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.227569 (+ 1881us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.227961 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.227970 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.227972 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.228029 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.228272 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.228274 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.228275 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.228458 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.228652 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.228859 (+ 207us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.229052 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"spinlock_wait_cycles":3328,"thread_start_us":138,"threads_started":1}
I20260202 15:04:20.229632 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.218591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59106)
0202 15:04:20.219240 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.219275 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.219842 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.220324 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.220329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.220700 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.221131 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.221145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.222164 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.222168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.222557 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.222565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.222777 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.223470 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.223493 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.224965 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.226988 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.226995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.227001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.227232 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.227498 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.227500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.227502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.227579 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.228020 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.228026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.228346 (+ 320us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.228516 (+ 170us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.228820 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"spinlock_wait_cycles":63232,"thread_start_us":194,"threads_started":1}
I20260202 15:04:20.230957 9992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260202 15:04:20.237962 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.231456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43290)
0202 15:04:20.231635 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.231651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.231752 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.231998 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.232000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.232171 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.232329 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.232334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.232985 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.232993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.233374 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.233380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.233494 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.233918 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.233930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.235082 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.236843 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.236846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.236848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.237075 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.237265 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.237267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.237269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.237310 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.237565 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.237567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.237644 (+ 77us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.237721 (+ 77us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.237810 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260202 15:04:20.237989 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.231537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.2.164.125:37549)
0202 15:04:20.231636 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.231641 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.231715 (+ 74us) server_negotiation.cc:408] Connection header received
0202 15:04:20.231869 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.231873 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.231914 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.231990 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.232434 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.232874 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.233475 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.233666 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.235197 (+ 1531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.235211 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.235213 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.235233 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.236720 (+ 1487us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.237148 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.237150 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.237152 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.237190 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.237375 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.237377 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.237378 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.237498 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.237564 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.237750 (+ 186us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.237862 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":43}
I20260202 15:04:20.240625 9992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Bootstrap starting.
I20260202 15:04:20.243175 9992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:20.243896 9992 log.cc:826] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:20.245724 9992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: No bootstrap required, opened a new log
I20260202 15:04:20.248250 9992 raft_consensus.cc:359] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } }
I20260202 15:04:20.248409 9992 raft_consensus.cc:385] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:20.248445 9992 raft_consensus.cc:740] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11e9461627f442096e6fdcba2f270b7, State: Initialized, Role: FOLLOWER
I20260202 15:04:20.248868 9992 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } }
I20260202 15:04:20.249390 9998 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } } }
I20260202 15:04:20.249538 9998 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:20.250094 9992 sys_catalog.cc:565] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: configured and running, proceeding with master startup.
W20260202 15:04:20.254601 10009 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:04:20.254691 10009 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:04:20.256322 9924 master_runner.cc:428] Detected that this master f11e9461627f442096e6fdcba2f270b7 is joining an existing cluster
I20260202 15:04:20.256373 9924 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.164.125:37549
I20260202 15:04:20.266438 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.258559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.2.164.126:45883)
0202 15:04:20.258674 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.258677 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.259289 (+ 612us) server_negotiation.cc:408] Connection header received
0202 15:04:20.259391 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.259393 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.259439 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.259555 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.260616 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.261466 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.262138 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.262692 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.263191 (+ 499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.263210 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.263213 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.263243 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.264855 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.265332 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.265336 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.265338 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.265390 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.265632 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.265635 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.265637 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.265797 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.265904 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.266168 (+ 264us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.266273 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260202 15:04:20.266448 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.258462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59108)
0202 15:04:20.259173 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.259188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.259298 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.259569 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.259572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.259839 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.260473 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.260482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.261596 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.261600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.262012 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.262019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.262160 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.262729 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.262745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.263052 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.265027 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.265031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.265035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.265229 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.265481 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.265484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.265487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.265538 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.265895 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.265898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.265997 (+ 99us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.266109 (+ 112us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.266252 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":110,"threads_started":1}
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260202 15:04:20.281558 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.271426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43296)
0202 15:04:20.271859 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.271905 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.272085 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.272419 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.272425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.273076 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.273369 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.273378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.274355 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.274359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.274924 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.274933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.275049 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.275565 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.275582 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.277215 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.279642 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.279652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.279658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.279964 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.280249 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.280255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.280314 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.280763 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.280808 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.280979 (+ 171us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.281178 (+ 199us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.281348 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
I20260202 15:04:20.281725 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.258939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.2.164.125:37549)
0202 15:04:20.259048 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.259051 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.259549 (+ 498us) server_negotiation.cc:408] Connection header received
0202 15:04:20.259693 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.259698 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.259743 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.259838 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.260664 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.261375 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.262216 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.262397 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.278096 (+ 15699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.278112 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.278115 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.278265 (+ 150us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.280093 (+ 1828us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280637 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.280641 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.280643 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.280685 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280922 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.280925 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.280927 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.281104 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.281191 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.281463 (+ 272us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.281563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260202 15:04:20.282280 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.258884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43294)
0202 15:04:20.259488 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.259500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.259593 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.259880 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.259884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.260201 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.260509 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.260518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.261495 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.261499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.262085 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.262094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.262492 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.277478 (+ 14986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.277491 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.277955 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.280200 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.280207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.280533 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.280782 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.280788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.280832 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.281206 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.281209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.281291 (+ 82us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.281395 (+ 104us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.281513 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":84,"threads_started":1}
I20260202 15:04:20.282922 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.271470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.2.164.125:37549)
0202 15:04:20.271788 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.271792 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.272172 (+ 380us) server_negotiation.cc:408] Connection header received
0202 15:04:20.272233 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.272235 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.272278 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.272344 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.273505 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.274241 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.275054 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.275204 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.277371 (+ 2167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.277403 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.277407 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.277433 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.279517 (+ 2084us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280085 (+ 568us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.280089 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.280091 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.280140 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.280430 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.280437 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.280440 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.280641 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.280757 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.282643 (+ 1886us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.282760 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260202 15:04:20.284216 9449 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.2.164.125:37549: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f11e9461627f442096e6fdcba2f270b7"
member_type: NON_VOTER
last_known_addr {
host: "127.2.164.125"
port: 37549
}
attrs {
promote: true
}
}
dest_uuid: "94296279378342b0afd24fa731eebc9f"
cas_config_opid_index: -1
I20260202 15:04:20.285148 9449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } }
I20260202 15:04:20.286861 10021 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f11e9461627f442096e6fdcba2f270b7 (127.2.164.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } }
I20260202 15:04:20.287444 9966 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:20.288410 10021 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.2.164.125:37549
I20260202 15:04:20.288528 9966 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Refusing update from remote peer 94296279378342b0afd24fa731eebc9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260202 15:04:20.288626 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } }
I20260202 15:04:20.288882 9924 master.cc:561] Master@127.2.164.125:37549 shutting down...
I20260202 15:04:20.288882 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:20.289264 10021 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:04:20.291133 9436 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260202 15:04:20.292331 9436 consensus_peers.cc:597] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f -> Peer f11e9461627f442096e6fdcba2f270b7 (127.2.164.125:37549): Couldn't send request to peer f11e9461627f442096e6fdcba2f270b7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:04:20.293346 9924 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:20.293628 9924 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:20.293756 9924 tablet_replica.cc:333] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: stopping tablet replica
WARNING: no policy specified for oryx/127.2.164.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.124@KRBTEST.COM" created.
I20260202 15:04:20.298669 9924 master.cc:583] Master@127.2.164.125:37549 shutdown complete.
I20260202 15:04:20.298727 9924 master_runner.cc:305] Clearing existing system tablet
I20260202 15:04:20.298985 9924 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:20.299043 9924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:20.300431 9924 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260202 15:04:20.301293 10027 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:20.301517 9924 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:20.301663 9924 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260202 15:04:20.301700 9924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
uuid: "f11e9461627f442096e6fdcba2f270b7"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "cf7c6c596c83814d4806eeb1aa744b66"
server_key_iv: "dd6ab20375c5036bdc7f2dc0baed8cf1"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.303393 9924 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:20.305176 9924 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260202 15:04:20.305275 9924 log.cc:1199] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260202 15:04:20.305495 9924 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Deleting consensus metadata
I20260202 15:04:20.305670 9924 master_runner.cc:315] Copying system tablet from 127.2.164.126:45883
I20260202 15:04:20.306849 9924 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: tablet copy: Beginning tablet copy session from remote peer at address 127.2.164.126:45883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:20.317673 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.307526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.2.164.126:45883)
0202 15:04:20.307758 (+ 232us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.307762 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.307778 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:04:20.307920 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.307922 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.307968 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.308040 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.308746 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.309439 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.310346 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.310477 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.311383 (+ 906us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.311411 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.311414 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.311441 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.315753 (+ 4312us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.316272 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.316278 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.316283 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.316344 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.316610 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.316617 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.316621 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.316839 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.316951 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.317374 (+ 423us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.317486 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260202 15:04:20.317684 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.307360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59120)
0202 15:04:20.307681 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.307698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.307808 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.308084 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.308087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.308287 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.308629 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.308638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.309553 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.309557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.310210 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.310220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.310367 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.310941 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.310955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.311244 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.315905 (+ 4661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.315910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.315913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.316153 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.316449 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.316453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.316455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.316514 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.317113 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.317116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.317199 (+ 83us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.317318 (+ 119us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.317480 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":116,"threads_started":1}
I20260202 15:04:20.318593 9472 tablet_copy_service.cc:140] P 94296279378342b0afd24fa731eebc9f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f11e9461627f442096e6fdcba2f270b7 ({username='oryx', principal='oryx/127.2.164.125@KRBTEST.COM'} at 127.0.0.1:59120)
I20260202 15:04:20.318768 9472 tablet_copy_service.cc:161] P 94296279378342b0afd24fa731eebc9f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f11e9461627f442096e6fdcba2f270b7 at {username='oryx', principal='oryx/127.2.164.125@KRBTEST.COM'} at 127.0.0.1:59120: session id = f11e9461627f442096e6fdcba2f270b7-00000000000000000000000000000000
I20260202 15:04:20.320173 9472 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:04:20.321812 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260202 15:04:20.323571 9924 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: tablet copy: Starting download of 0 data blocks...
I20260202 15:04:20.323809 9924 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: tablet copy: Starting download of 1 WAL segments...
WARNING: no policy specified for HTTP/127.2.164.124@KRBTEST.COM; defaulting to no policy
I20260202 15:04:20.325906 9924 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.124@KRBTEST.COM" created.
I20260202 15:04:20.327984 9472 tablet_copy_service.cc:342] P 94296279378342b0afd24fa731eebc9f: Request end of tablet copy session f11e9461627f442096e6fdcba2f270b7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.164.125@KRBTEST.COM'} at 127.0.0.1:59120
I20260202 15:04:20.328141 9472 tablet_copy_service.cc:434] P 94296279378342b0afd24fa731eebc9f: ending tablet copy session f11e9461627f442096e6fdcba2f270b7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f11e9461627f442096e6fdcba2f270b7
W20260202 15:04:20.333142 9924 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260202 15:04:20.335659 9924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:20.337355 10037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:20.337551 10036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:20.337823 10039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:20.337962 9924 server_base.cc:1060] running on GCE node
I20260202 15:04:20.338117 9924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:20.338429 9924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:20.339573 9924 hybrid_clock.cc:648] HybridClock initialized: now 1770044660339564 us; error 30 us; skew 500 ppm
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:20.344144 9924 init.cc:377] Logged in from keytab as oryx/127.2.164.125@KRBTEST.COM (short username oryx)
I20260202 15:04:20.344306 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:41495
--webserver_interface=127.2.164.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260202 15:04:20.344676 9924 webserver.cc:492] Webserver started at http://127.2.164.125:38745/ using document root <none> and password file <none>
I20260202 15:04:20.344892 9924 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:20.344935 9924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:20.346318 9924 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:20.347370 10047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:20.347606 9924 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:20.347697 9924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
uuid: "f11e9461627f442096e6fdcba2f270b7"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "cf7c6c596c83814d4806eeb1aa744b66"
server_key_iv: "dd6ab20375c5036bdc7f2dc0baed8cf1"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.347800 9924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:20.352747 9924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:20.354667 9924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:20.359589 9924 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:37549
I20260202 15:04:20.359809 10099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.125:37549 every 8 connection(s)
I20260202 15:04:20.360234 9924 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-1/data/info.pb
I20260202 15:04:20.361783 10100 sys_catalog.cc:263] Verifying existing consensus state
I20260202 15:04:20.362529 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Bootstrap starting.
I20260202 15:04:20.370805 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:04:20.371152 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: Bootstrap complete.
I20260202 15:04:20.371718 10100 raft_consensus.cc:359] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } }
I20260202 15:04:20.371867 10100 raft_consensus.cc:740] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f11e9461627f442096e6fdcba2f270b7, State: Initialized, Role: LEARNER
I20260202 15:04:20.371982 10100 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } }
I20260202 15:04:20.372337 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } }
I20260202 15:04:20.372483 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: LEARNER
I20260202 15:04:20.372510 10100 sys_catalog.cc:565] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:20.375608 10114 catalog_manager.cc:1269] Loaded cluster ID: dbfff5e6ac374cf4b4b93b06201d8cd4
I20260202 15:04:20.375701 10114 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: loading cluster ID for follower catalog manager: success
I20260202 15:04:20.377394 10114 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: acquiring CA information for follower catalog manager: success
I20260202 15:04:20.378504 10114 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260202 15:04:20.474748 10045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:20.475103 10045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:20.475212 10045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:20.478749 10045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:20.478854 10045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:20.478899 10045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:20.478955 10045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:20.479012 10045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:20.483495 10045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:41495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.164.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:20.484673 10045 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:20.485680 10045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:20.491452 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:20.491451 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:20.491500 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:20.491814 10045 server_base.cc:1060] running on GCE node
I20260202 15:04:20.492304 10045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:20.493290 10045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:20.494442 10045 hybrid_clock.cc:648] HybridClock initialized: now 1770044660494419 us; error 38 us; skew 500 ppm
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:20.497153 10045 init.cc:377] Logged in from keytab as oryx/127.2.164.124@KRBTEST.COM (short username oryx)
I20260202 15:04:20.498472 10045 webserver.cc:492] Webserver started at http://127.2.164.124:43059/ using document root <none> and password file <none>
I20260202 15:04:20.499131 10045 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:20.499202 10045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:20.499399 10045 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:20.501499 10045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/instance:
uuid: "7aea978ffa0f48749984cdc07f255126"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "5e1ed688687aed4ca868ce7b080ba802"
server_key_iv: "abb62bdb3b0334a8e609f4d8689a998f"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.501955 10045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal/instance:
uuid: "7aea978ffa0f48749984cdc07f255126"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "5e1ed688687aed4ca868ce7b080ba802"
server_key_iv: "abb62bdb3b0334a8e609f4d8689a998f"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.505344 10045 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:20.507552 10129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:20.508577 10045 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:20.508682 10045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
uuid: "7aea978ffa0f48749984cdc07f255126"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "5e1ed688687aed4ca868ce7b080ba802"
server_key_iv: "abb62bdb3b0334a8e609f4d8689a998f"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.508761 10045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:20.520584 9645 heartbeater.cc:499] Master 127.2.164.126:45883 was elected leader, sending a full tablet report...
I20260202 15:04:20.520816 10045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:20.523778 10045 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:20.523972 10045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:20.531978 10045 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:41495
I20260202 15:04:20.531984 10181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.124:41495 every 8 connection(s)
I20260202 15:04:20.533031 10045 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/info.pb
I20260202 15:04:20.535995 10182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:20.539806 10182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
I20260202 15:04:20.542030 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 10045
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260202 15:04:20.556156 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.541948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.2.164.126:45883)
0202 15:04:20.542110 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.542115 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.543003 (+ 888us) server_negotiation.cc:408] Connection header received
0202 15:04:20.544267 (+ 1264us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.544271 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.544332 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.544412 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.546446 (+ 2034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.547462 (+ 1016us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.548573 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.548780 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.551851 (+ 3071us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.551872 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.551874 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.551903 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.553675 (+ 1772us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.554237 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.554241 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.554242 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.554298 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.554745 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.554749 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.554750 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.554958 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.555079 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.555828 (+ 749us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.555961 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260202 15:04:20.556910 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.542135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59122)
0202 15:04:20.542792 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.542877 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.544006 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.544576 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.544597 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.545057 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.546280 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.546295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.547614 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.547619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.548405 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.548415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.549138 (+ 723us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.549804 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.549846 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.551641 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.553802 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.553809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.553821 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.554135 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.554394 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.554397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.554399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.554523 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.555118 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.555127 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.555479 (+ 352us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.555724 (+ 245us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.556000 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":163,"threads_started":1}
I20260202 15:04:20.556910 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.545497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.2.164.124:41495)
0202 15:04:20.545885 (+ 388us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.545891 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.545913 (+ 22us) server_negotiation.cc:408] Connection header received
0202 15:04:20.545951 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.545958 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.546154 (+ 196us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.546307 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.547459 (+ 1152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.548516 (+ 1057us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.549358 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.549604 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.552246 (+ 2642us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.552272 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.552286 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.552317 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.554567 (+ 2250us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.555152 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.555158 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.555159 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.555216 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.555437 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.555440 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.555442 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.555671 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.555859 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.556140 (+ 281us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.556268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"spinlock_wait_cycles":2944,"thread_start_us":124,"threads_started":1}
I20260202 15:04:20.558202 10182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260202 15:04:20.565750 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.558708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43298)
0202 15:04:20.559096 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.559108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.559169 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.559417 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.559419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.559547 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.559718 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.559723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.560598 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.560601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.560965 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.560971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.561126 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.561584 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.561596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.562775 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.564500 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.564502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.564504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.564659 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.564925 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.564930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.564936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.564981 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.565363 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.565365 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.565441 (+ 76us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.565533 (+ 92us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.565629 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260202 15:04:20.565888 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.558962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.2.164.125:37549)
0202 15:04:20.559223 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.559227 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.559239 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:20.559274 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.559276 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.559315 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.559393 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.559819 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.560507 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.561124 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.561254 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.562898 (+ 1644us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.562914 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.562916 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.562938 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.564408 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.564739 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.564742 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.564743 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.564783 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.565104 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.565106 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.565108 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.565245 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.565330 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.565600 (+ 270us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.565697 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260202 15:04:20.566598 10182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } has no permanent_uuid. Determining permanent_uuid...
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260202 15:04:20.574007 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.567104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44414)
0202 15:04:20.567240 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.567256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.567364 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.567575 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.567577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.567723 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.567865 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.567870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.568514 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.568520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.568852 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.568858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.568973 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.569383 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.569394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.570494 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.572372 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.572375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.572377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.572525 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.572763 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.572766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.572767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.572816 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.573392 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.573395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.573465 (+ 70us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.573599 (+ 134us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.573771 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260202 15:04:20.574008 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.567138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.2.164.124:41495)
0202 15:04:20.567240 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.567246 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.567320 (+ 74us) server_negotiation.cc:408] Connection header received
0202 15:04:20.567446 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.567449 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.567490 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.567566 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.567969 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.568417 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.568955 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.569218 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.570599 (+ 1381us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.570613 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.570615 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.570636 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.572193 (+ 1557us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.572608 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.572611 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.572612 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.572652 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.573161 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.573164 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.573166 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.573298 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.573387 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.573640 (+ 253us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.573834 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":41}
I20260202 15:04:20.576961 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Bootstrap starting.
I20260202 15:04:20.579491 10182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:20.580205 10182 log.cc:826] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:20.581986 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: No bootstrap required, opened a new log
I20260202 15:04:20.584905 10182 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } }
I20260202 15:04:20.585152 10182 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:20.585229 10182 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aea978ffa0f48749984cdc07f255126, State: Initialized, Role: FOLLOWER
I20260202 15:04:20.585635 10182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } }
I20260202 15:04:20.586103 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } } }
I20260202 15:04:20.586241 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:20.586715 10182 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: configured and running, proceeding with master startup.
W20260202 15:04:20.591418 10199 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:04:20.591511 10199 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:04:20.595371 10045 master_runner.cc:428] Detected that this master 7aea978ffa0f48749984cdc07f255126 is joining an existing cluster
I20260202 15:04:20.595477 10045 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.164.124:41495
I20260202 15:04:20.602893 10188 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:20.603121 10188 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } }
I20260202 15:04:20.604074 10188 leader_election.cc:290] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883), f11e9461627f442096e6fdcba2f270b7 (127.2.164.125:37549)
I20260202 15:04:20.604516 9462 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aea978ffa0f48749984cdc07f255126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94296279378342b0afd24fa731eebc9f" is_pre_election: true
I20260202 15:04:20.604583 10074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aea978ffa0f48749984cdc07f255126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11e9461627f442096e6fdcba2f270b7" is_pre_election: true
I20260202 15:04:20.604768 10074 raft_consensus.cc:1813] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Handling vote request from an unknown peer 7aea978ffa0f48749984cdc07f255126
I20260202 15:04:20.604897 10074 raft_consensus.cc:2410] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 7aea978ffa0f48749984cdc07f255126 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260202 15:04:20.605343 10130 leader_election.cc:304] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7aea978ffa0f48749984cdc07f255126; no voters: 94296279378342b0afd24fa731eebc9f, f11e9461627f442096e6fdcba2f270b7
I20260202 15:04:20.605624 10188 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:20.622870 10188 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260202 15:04:20.629963 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.625266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59148)
0202 15:04:20.625383 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.625397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.625471 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.627129 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.627132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.627147 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:20.627360 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.627371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.628566 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.628572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.629521 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.629531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.629615 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.629685 (+ 70us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.629734 (+ 49us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.629819 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260202 15:04:20.630575 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.598210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.2.164.124:41495)
0202 15:04:20.598350 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.598354 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.598760 (+ 406us) server_negotiation.cc:408] Connection header received
0202 15:04:20.598902 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.598905 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.598946 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.599026 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.621857 (+ 22831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.622569 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.623366 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.623539 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.626269 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.626286 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.626289 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.626310 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.628486 (+ 2176us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.628876 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.628879 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.628881 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.628918 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.629389 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.629392 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.629394 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.629548 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.629613 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.630303 (+ 690us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.630411 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260202 15:04:20.631675 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.598463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.2.164.126:45883)
0202 15:04:20.598606 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.598613 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.598664 (+ 51us) server_negotiation.cc:408] Connection header received
0202 15:04:20.598898 (+ 234us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.598906 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.598966 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.599064 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.624194 (+ 25130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.625046 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.626172 (+ 1126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.626370 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.627851 (+ 1481us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.627869 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.627872 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.627899 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.629528 (+ 1629us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.629967 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.629971 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.629974 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.630032 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.630544 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.630547 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.630550 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.630709 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.630776 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.631365 (+ 589us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.631477 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260202 15:04:20.631839 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.598140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59134)
0202 15:04:20.598580 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.598596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.598739 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.599189 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.599192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.622726 (+ 23534us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.624029 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.624037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.625188 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.625192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.626034 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.626045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.626485 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.626923 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.626940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.627718 (+ 778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.629662 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.629665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.629667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.629861 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.630136 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.630139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.630141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.630434 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.630846 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.630850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.630923 (+ 73us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.631549 (+ 626us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.631669 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":147,"threads_started":1}
I20260202 15:04:20.632215 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.598140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44422)
0202 15:04:20.598670 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.598681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.598806 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.599070 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.599073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.621295 (+ 22222us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.621669 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.621682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.622690 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.622694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.623244 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.623252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.623647 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.625701 (+ 2054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.625718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.626149 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.628591 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.628595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.628597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.628786 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.629038 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.629041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.629043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.629089 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.629672 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.629674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.629746 (+ 72us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.631120 (+ 1374us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.631270 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":233,"thread_start_us":93,"threads_started":1}
I20260202 15:04:20.633553 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.625443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.2.164.126:45883)
0202 15:04:20.626876 (+ 1433us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.626882 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.626897 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:04:20.626950 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.626953 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.627006 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.627473 (+ 467us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:20.627587 (+ 114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.628435 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.630534 (+ 2099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.631312 (+ 778us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.631384 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.631438 (+ 54us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.631511 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1349,"thread_start_us":119,"threads_started":1}
I20260202 15:04:20.633814 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.622295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.2.164.125:37549)
0202 15:04:20.622440 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.622445 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.623375 (+ 930us) server_negotiation.cc:408] Connection header received
0202 15:04:20.623592 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.623596 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.623653 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.623705 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.624727 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.625509 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.626304 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.626484 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.629886 (+ 3402us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.629909 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.629912 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.629942 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.632158 (+ 2216us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.632599 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.632602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.632604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.632653 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.632899 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.632902 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.632904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.633096 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.633180 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.633517 (+ 337us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.633659 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:04:20.634251 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.621789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43304)
0202 15:04:20.623256 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.623275 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.623464 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.623836 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.623838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.624321 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.624574 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.624581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.625643 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.625646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.626186 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.626194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.626609 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.627266 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.627280 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.629283 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.632279 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.632282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.632284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.632483 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.632760 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.632763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.632765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.632813 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.633252 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.633254 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.633337 (+ 83us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.633936 (+ 599us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.634097 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":131,"threads_started":1}
I20260202 15:04:20.638167 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.635131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43308)
0202 15:04:20.635361 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.635375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.635451 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.635766 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.635769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.635788 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:20.636018 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.636045 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.636955 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.636959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.637793 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.637800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.637919 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.637938 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.637989 (+ 51us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.638050 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260202 15:04:20.638729 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.635376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.2.164.125:37549)
0202 15:04:20.635512 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.635516 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.635531 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:04:20.635574 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.635580 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.635642 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.635757 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:20.636135 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.636837 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.637906 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.638382 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.638539 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.638593 (+ 54us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.638645 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044659, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260202 15:04:20.644934 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.636426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.2.164.124:41495)
0202 15:04:20.636529 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.636533 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.636753 (+ 220us) server_negotiation.cc:408] Connection header received
0202 15:04:20.636882 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.636885 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.636937 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.637041 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.637596 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.638284 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.639055 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.639235 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.641289 (+ 2054us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.641305 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.641308 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.641331 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.643405 (+ 2074us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.643839 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.643843 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.643845 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.643899 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.644167 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.644171 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.644173 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.644330 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.644436 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.644681 (+ 245us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.644785 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260202 15:04:20.644946 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.636341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44436)
0202 15:04:20.636662 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.636675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.636763 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.637047 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.637050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.637254 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.637476 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.637483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.638411 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.638415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.638913 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.638922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.639043 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.639627 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.639643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.641169 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.643533 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.643537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.643539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.643753 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.644004 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.644008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.644011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.644075 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.644425 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.644428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.644521 (+ 93us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.644631 (+ 110us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.644779 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
I20260202 15:04:20.646118 9451 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.2.164.124:41495: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7aea978ffa0f48749984cdc07f255126"
member_type: NON_VOTER
last_known_addr {
host: "127.2.164.124"
port: 41495
}
attrs {
promote: true
}
}
dest_uuid: "94296279378342b0afd24fa731eebc9f"
cas_config_opid_index: 5
I20260202 15:04:20.646740 9451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } }
I20260202 15:04:20.647886 10022 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7aea978ffa0f48749984cdc07f255126 (127.2.164.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.649312 10022 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.2.164.124:41495
I20260202 15:04:20.649487 10022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.649569 10022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:20.649894 10156 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Refusing update from remote peer 94296279378342b0afd24fa731eebc9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260202 15:04:20.649987 10045 master.cc:561] Master@127.2.164.124:41495 shutting down...
I20260202 15:04:20.650585 10074 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Refusing update from remote peer 94296279378342b0afd24fa731eebc9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260202 15:04:20.651517 10021 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:20.653431 10021 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260202 15:04:20.653577 9438 consensus_peers.cc:597] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f -> Peer 7aea978ffa0f48749984cdc07f255126 (127.2.164.124:41495): Couldn't send request to peer 7aea978ffa0f48749984cdc07f255126. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260202 15:04:20.654860 10045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:20.655117 10045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:20.655212 10045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: stopping tablet replica
I20260202 15:04:20.655279 10074 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7aea978ffa0f48749984cdc07f255126 (127.2.164.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.657311 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94296279378342b0afd24fa731eebc9f. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.657423 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: LEARNER
I20260202 15:04:20.657524 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: true } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.657590 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: LEARNER
I20260202 15:04:20.657915 10022 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: attempting to promote NON_VOTER f11e9461627f442096e6fdcba2f270b7 to VOTER
I20260202 15:04:20.658337 10022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } }
I20260202 15:04:20.659766 10074 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 LEARNER]: Refusing update from remote peer 94296279378342b0afd24fa731eebc9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260202 15:04:20.660197 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.659168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44444)
0202 15:04:20.659398 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.659462 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260202 15:04:20.660176 10022 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260202 15:04:20.660897 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.660629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44458)
0202 15:04:20.660776 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.660822 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260202 15:04:20.661077 9438 consensus_peers.cc:597] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f -> Peer 7aea978ffa0f48749984cdc07f255126 (127.2.164.124:41495): Couldn't send request to peer 7aea978ffa0f48749984cdc07f255126. Status: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260202 15:04:20.661819 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.661578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44460)
0202 15:04:20.661708 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.661757 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260202 15:04:20.662032 9449 master.cc:679] Network error: unable to get registration information for peer 7aea978ffa0f48749984cdc07f255126: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
I20260202 15:04:20.662528 10021 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f11e9461627f442096e6fdcba2f270b7 (127.2.164.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.663076 10074 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f11e9461627f442096e6fdcba2f270b7 (127.2.164.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.663992 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.664108 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:20.664260 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.664347 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:20.665139 10211 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.665246 10211 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:20.673628 10045 master.cc:583] Master@127.2.164.124:41495 shutdown complete.
I20260202 15:04:20.673700 10045 master_runner.cc:305] Clearing existing system tablet
I20260202 15:04:20.673964 10045 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:20.674042 10045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:20.675472 10045 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260202 15:04:20.676016 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.675714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44472)
0202 15:04:20.675864 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.675928 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260202 15:04:20.676241 9449 master.cc:679] Network error: unable to get registration information for peer 7aea978ffa0f48749984cdc07f255126: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
I20260202 15:04:20.676417 10214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:20.676628 10045 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:20.676761 10045 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260202 15:04:20.676805 10045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
uuid: "7aea978ffa0f48749984cdc07f255126"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "5e1ed688687aed4ca868ce7b080ba802"
server_key_iv: "abb62bdb3b0334a8e609f4d8689a998f"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.678678 10045 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:20.680550 10045 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260202 15:04:20.680645 10045 log.cc:1199] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260202 15:04:20.680928 10045 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Deleting consensus metadata
I20260202 15:04:20.681141 10045 master_runner.cc:315] Copying system tablet from 127.2.164.126:45883
I20260202 15:04:20.682322 10045 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: tablet copy: Beginning tablet copy session from remote peer at address 127.2.164.126:45883
I20260202 15:04:20.690104 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.682856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59162)
0202 15:04:20.683129 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.683142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.683243 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.683509 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.683511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.683917 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:20.684135 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.684142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.685270 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.685277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.685865 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.685875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.685982 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.686585 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:20.686599 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:20.686896 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:20.688719 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.688722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.688724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.688892 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.689188 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:20.689191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:20.689192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:20.689238 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:20.689599 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:20.689601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:20.689675 (+ 74us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.689754 (+ 79us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.689914 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260202 15:04:20.690133 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.682922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.2.164.126:45883)
0202 15:04:20.683081 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.683087 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.683213 (+ 126us) server_negotiation.cc:408] Connection header received
0202 15:04:20.683331 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.683335 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.683391 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.683487 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:20.684257 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.685111 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.685984 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.686189 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.687011 (+ 822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:20.687030 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:20.687034 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:20.687064 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:20.688603 (+ 1539us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.688983 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.688986 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.688989 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.689071 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:20.689322 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:20.689325 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:20.689327 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:20.689506 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:20.689609 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.689828 (+ 219us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.689963 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260202 15:04:20.690529 9472 tablet_copy_service.cc:140] P 94296279378342b0afd24fa731eebc9f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7aea978ffa0f48749984cdc07f255126 ({username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:59162)
I20260202 15:04:20.690629 9472 tablet_copy_service.cc:161] P 94296279378342b0afd24fa731eebc9f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7aea978ffa0f48749984cdc07f255126 at {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:59162: session id = 7aea978ffa0f48749984cdc07f255126-00000000000000000000000000000000
I20260202 15:04:20.691408 9472 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Tablet Copy: opened 0 blocks and 1 log segments
I20260202 15:04:20.692626 10045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260202 15:04:20.694909 10045 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: tablet copy: Starting download of 0 data blocks...
I20260202 15:04:20.695117 10045 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: tablet copy: Starting download of 1 WAL segments...
I20260202 15:04:20.696152 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.695880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44482)
0202 15:04:20.696014 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.696071 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260202 15:04:20.696375 9449 master.cc:679] Network error: unable to get registration information for peer 7aea978ffa0f48749984cdc07f255126: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
I20260202 15:04:20.697057 10045 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260202 15:04:20.698616 9472 tablet_copy_service.cc:342] P 94296279378342b0afd24fa731eebc9f: Request end of tablet copy session 7aea978ffa0f48749984cdc07f255126-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:59162
I20260202 15:04:20.698704 9472 tablet_copy_service.cc:434] P 94296279378342b0afd24fa731eebc9f: ending tablet copy session 7aea978ffa0f48749984cdc07f255126-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7aea978ffa0f48749984cdc07f255126
W20260202 15:04:20.703475 10045 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260202 15:04:20.705806 10045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:20.707636 10221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:20.707695 10222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:20.707866 10045 server_base.cc:1060] running on GCE node
W20260202 15:04:20.707736 10224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:20.708156 10045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:20.708477 10045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:20.709718 10045 hybrid_clock.cc:648] HybridClock initialized: now 1770044660709703 us; error 34 us; skew 500 ppm
Feb 02 15:04:20 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044660, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:20.711472 10045 init.cc:377] Logged in from keytab as oryx/127.2.164.124@KRBTEST.COM (short username oryx)
I20260202 15:04:20.712026 10045 webserver.cc:492] Webserver started at http://127.2.164.124:34085/ using document root <none> and password file <none>
I20260202 15:04:20.712248 10045 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:20.712303 10045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:20.713512 10045 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:20.714515 10231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:20.714794 10045 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:20.714885 10045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
uuid: "7aea978ffa0f48749984cdc07f255126"
format_stamp: "Formatted at 2026-02-02 15:04:20 on dist-test-slave-6zzp"
server_key: "5e1ed688687aed4ca868ce7b080ba802"
server_key_iv: "abb62bdb3b0334a8e609f4d8689a998f"
server_key_version: "encryptionkey@0"
I20260202 15:04:20.714983 10045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:20.727002 10045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:20.728884 10045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:20.732545 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.732225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44494)
0202 15:04:20.732397 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.732450 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260202 15:04:20.732766 9449 master.cc:679] Network error: unable to get registration information for peer 7aea978ffa0f48749984cdc07f255126: Client connection negotiation failed: client connection to 127.2.164.124:41495: connect: Connection refused (error 111)
I20260202 15:04:20.734551 10045 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:41495
I20260202 15:04:20.734894 10283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.124:41495 every 8 connection(s)
I20260202 15:04:20.735355 10045 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/master-2/data/info.pb
I20260202 15:04:20.736935 10284 sys_catalog.cc:263] Verifying existing consensus state
I20260202 15:04:20.737792 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Bootstrap starting.
I20260202 15:04:20.746739 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:04:20.747052 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Bootstrap complete.
I20260202 15:04:20.747663 10284 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } }
I20260202 15:04:20.747800 10284 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7aea978ffa0f48749984cdc07f255126, State: Initialized, Role: LEARNER
I20260202 15:04:20.747949 10284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } }
I20260202 15:04:20.748301 10287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:20.748397 10287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: This master's current role is: LEARNER
I20260202 15:04:20.748477 10284 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:20.752447 10298 catalog_manager.cc:1269] Loaded cluster ID: dbfff5e6ac374cf4b4b93b06201d8cd4
I20260202 15:04:20.752511 10298 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: loading cluster ID for follower catalog manager: success
I20260202 15:04:20.754313 10298 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: acquiring CA information for follower catalog manager: success
I20260202 15:04:20.755510 10298 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260202 15:04:20.755609 9781 heartbeater.cc:499] Master 127.2.164.126:45883 was elected leader, sending a full tablet report...
I20260202 15:04:20.804586 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.801595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44500)
0202 15:04:20.801778 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:20.801793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:20.801881 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:20.802335 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:20.802339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:20.802357 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:20.802561 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.802567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.803521 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.803526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:20.804148 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:20.804156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.804292 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.804315 (+ 23us) client_negotiation.cc:769] Sending connection context
0202 15:04:20.804367 (+ 52us) client_negotiation.cc:240] Negotiation successful
0202 15:04:20.804437 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260202 15:04:20.805234 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:20.801791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.2.164.124:41495)
0202 15:04:20.802074 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:20.802078 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:20.802097 (+ 19us) server_negotiation.cc:408] Connection header received
0202 15:04:20.802139 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:20.802141 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:20.802194 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:20.802321 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:20.802687 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.803393 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:20.804278 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:20.804771 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:20.804979 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:20.805064 (+ 85us) server_negotiation.cc:300] Negotiation successful
0202 15:04:20.805135 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260202 15:04:20.981983 9917 heartbeater.cc:499] Master 127.2.164.126:45883 was elected leader, sending a full tablet report...
I20260202 15:04:21.075989 10258 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260202 15:04:21.076412 10287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94296279378342b0afd24fa731eebc9f. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:21.076553 10287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: This master's current role is: LEARNER
I20260202 15:04:21.507437 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: true } } }
I20260202 15:04:21.507627 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:21.508500 10301 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: attempting to promote NON_VOTER 7aea978ffa0f48749984cdc07f255126 to VOTER
I20260202 15:04:21.508910 10301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } }
I20260202 15:04:21.510406 10258 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 LEARNER]: Refusing update from remote peer 94296279378342b0afd24fa731eebc9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260202 15:04:21.510417 10074 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Refusing update from remote peer 94296279378342b0afd24fa731eebc9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260202 15:04:21.510847 10212 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:04:21.511086 10301 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260202 15:04:21.512811 10212 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7aea978ffa0f48749984cdc07f255126 (127.2.164.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.513339 10074 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7aea978ffa0f48749984cdc07f255126 (127.2.164.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.514467 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.514575 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:21.514696 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.514755 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:21.514518 10258 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7aea978ffa0f48749984cdc07f255126 (127.2.164.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.514858 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.514930 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:21.515240 10304 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.515347 10304 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:21.516096 10287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94296279378342b0afd24fa731eebc9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:21.516238 10287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:21.726704 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.716382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.2.164.125:37549)
0202 15:04:21.716655 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.716659 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.716674 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:04:21.716768 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.716770 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.716811 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.716893 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:21.717736 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.718277 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.718922 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.719063 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.723499 (+ 4436us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:21.723519 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:21.723521 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:21.723547 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:21.725203 (+ 1656us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.725589 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.725592 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.725593 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.725632 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.725818 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.725820 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.725821 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.725989 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:21.726091 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.726352 (+ 261us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.726458 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260202 15:04:21.730157 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.727706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59168)
0202 15:04:21.727970 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.727982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.728057 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.728308 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.728310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.728323 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:21.728500 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.728505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.729286 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.729288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.729850 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.729857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.729961 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.729976 (+ 15us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.730013 (+ 37us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.730056 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
I20260202 15:04:21.730572 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.727867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.2.164.126:45883)
0202 15:04:21.728114 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.728118 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.728131 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:21.728174 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.728176 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.728217 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.728291 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:21.728598 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.729191 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.729966 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.730317 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.730358 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.730415 (+ 57us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.730476 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260202 15:04:21.733922 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.731750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43336)
0202 15:04:21.731871 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.731890 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.731948 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.732143 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.732145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.732160 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:21.732293 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.732297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.732942 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.732944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.733620 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.733628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.733716 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.733729 (+ 13us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.733764 (+ 35us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.733802 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260202 15:04:21.734613 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.731799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.2.164.125:37549)
0202 15:04:21.731925 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.731928 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.731940 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:21.732023 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.732026 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.732061 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.732122 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:21.732377 (+ 255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.732855 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.733732 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.734382 (+ 650us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.734420 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.734470 (+ 50us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.734520 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260202 15:04:21.740141 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.737391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44506)
0202 15:04:21.737500 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.737513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.737582 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.737971 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.737974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.737994 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:21.738189 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.738194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.738945 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.738949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.739808 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.739816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.739928 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.739945 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.739989 (+ 44us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.740035 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260202 15:04:21.740641 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.737491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.2.164.124:41495)
0202 15:04:21.737768 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.737772 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.737789 (+ 17us) server_negotiation.cc:408] Connection header received
0202 15:04:21.737827 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.737830 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.737868 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.737931 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:21.738343 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.738841 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.739958 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.740412 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.740456 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.740498 (+ 42us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.740550 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260202 15:04:21.748790 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.742500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.2.164.124:41495)
0202 15:04:21.742611 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.742615 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.742665 (+ 50us) server_negotiation.cc:408] Connection header received
0202 15:04:21.742769 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.742772 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.742815 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.742899 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:21.743585 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.744034 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.744584 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.744703 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.745885 (+ 1182us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:21.745909 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:21.745912 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:21.745935 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:21.747316 (+ 1381us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.747852 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.747855 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.747856 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.747901 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.748139 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.748141 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.748142 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.748274 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:21.748378 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.748580 (+ 202us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.748682 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260202 15:04:21.752192 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.749681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59184)
0202 15:04:21.749957 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.749969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.750063 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.750319 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.750322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.750342 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:21.750519 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.750524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.751290 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.751293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.751888 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.751895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.751988 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.752002 (+ 14us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.752048 (+ 46us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.752095 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
I20260202 15:04:21.752676 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.749810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.2.164.126:45883)
0202 15:04:21.749974 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.749977 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.750030 (+ 53us) server_negotiation.cc:408] Connection header received
0202 15:04:21.750150 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.750153 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.750195 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.750263 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:21.750650 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.751188 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.752014 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.752439 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.752492 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.752536 (+ 44us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.752585 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260202 15:04:21.756068 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.753856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.0.0.1:43350)
0202 15:04:21.754002 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.754015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.754082 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.754327 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.754330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.754345 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:21.754480 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.754484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.755153 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.755156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.755744 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.755752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.755868 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.755886 (+ 18us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.755939 (+ 53us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.755984 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260202 15:04:21.756524 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.754029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.2.164.125:37549)
0202 15:04:21.754151 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.754155 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.754167 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:21.754206 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.754209 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.754246 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.754308 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:21.754578 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.755058 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.755880 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.756308 (+ 428us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.756345 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.756386 (+ 41us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.756431 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260202 15:04:21.759944 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.757716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.0.0.1:44526)
0202 15:04:21.757868 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.757881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.757948 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.758271 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.758274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.758294 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:21.758475 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.758481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.759135 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.759138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.759661 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.759668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.759757 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.759771 (+ 14us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.759813 (+ 42us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.759853 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260202 15:04:21.760344 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.757779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.2.164.124:41495)
0202 15:04:21.757895 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.757898 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.757912 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:21.758035 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.758037 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.758073 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.758165 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:21.758575 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.759035 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.759769 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.760162 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.760199 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.760236 (+ 37us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.760273 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260202 15:04:21.761781 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9516
I20260202 15:04:21.769244 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:38971
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:45883
--webserver_port=38255
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:21.882036 10311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:21.882282 10311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:21.882339 10311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:21.885864 10311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:21.885927 10311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:21.886008 10311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:21.890198 10311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:38971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=38255
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:21.891207 10311 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:21.892045 10311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:21.900058 10317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:21.900060 10319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:21.900058 10316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:21.900362 10311 server_base.cc:1060] running on GCE node
I20260202 15:04:21.900719 10311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:21.901360 10311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:21.902519 10311 hybrid_clock.cc:648] HybridClock initialized: now 1770044661902491 us; error 43 us; skew 500 ppm
Feb 02 15:04:21 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044661, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:21.905911 10311 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260202 15:04:21.907033 10311 webserver.cc:492] Webserver started at http://127.2.164.65:38255/ using document root <none> and password file <none>
I20260202 15:04:21.907596 10311 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:21.907656 10311 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:21.911154 10311 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:21.913781 10326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:21.915139 10311 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:04:21.915282 10311 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "604b79fe893d4460ad4c782af7e87396"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "29fd5dac4d168ba37d06259bc4671f3e"
server_key_iv: "2726f4f14dcf920fdfab0ad89406ef35"
server_key_version: "encryptionkey@0"
I20260202 15:04:21.915786 10311 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:21.932273 10311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:21.935690 10311 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:21.935896 10311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:21.936483 10311 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:21.937492 10311 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:21.937573 10311 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:21.937644 10311 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:21.937705 10311 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:21.947764 10311 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:38971
I20260202 15:04:21.947795 10439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:38971 every 8 connection(s)
I20260202 15:04:21.948813 10311 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:21.955397 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 10311
I20260202 15:04:21.955580 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9652
Feb 02 15:04:21 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044661, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
Feb 02 15:04:21 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044661, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Feb 02 15:04:21 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044661, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260202 15:04:21.963292 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:32967
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:45883
--webserver_port=42519
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260202 15:04:21.966218 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.952118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34831 (local address 127.2.164.125:37549)
0202 15:04:21.952244 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.952249 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.952463 (+ 214us) server_negotiation.cc:408] Connection header received
0202 15:04:21.953098 (+ 635us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.953102 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.953152 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.953244 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:21.954519 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.955013 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.957355 (+ 2342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.958158 (+ 803us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.961314 (+ 3156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:21.961337 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:21.961341 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:21.961375 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:21.963193 (+ 1818us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.964356 (+ 1163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.964361 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.964365 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.964429 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.964829 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.964833 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.964834 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.965095 (+ 261us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:21.965170 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.965910 (+ 740us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.966056 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260202 15:04:21.966369 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.950715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:40421 (local address 127.2.164.124:41495)
0202 15:04:21.950843 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.950847 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.951555 (+ 708us) server_negotiation.cc:408] Connection header received
0202 15:04:21.952812 (+ 1257us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.952822 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.952870 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.952960 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:21.954450 (+ 1490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.955216 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.956692 (+ 1476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.956889 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.960309 (+ 3420us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:21.960336 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:21.960343 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:21.960380 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:21.964218 (+ 3838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.964849 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.964853 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.964855 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.964902 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.965512 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.965515 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.965517 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.965685 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:21.965755 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.966114 (+ 359us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.966228 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:04:21.966876 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.950678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:47623 (local address 127.2.164.126:45883)
0202 15:04:21.950830 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:21.950835 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:21.951553 (+ 718us) server_negotiation.cc:408] Connection header received
0202 15:04:21.952677 (+ 1124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:21.952681 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:21.952739 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:21.952842 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:21.954521 (+ 1679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.955270 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.956692 (+ 1422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.956903 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.960713 (+ 3810us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:21.960735 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:21.960741 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:21.960791 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:21.962733 (+ 1942us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.965491 (+ 2758us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.965495 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.965497 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.965557 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:21.965852 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:21.965856 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:21.965858 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:21.966074 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:21.966167 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:21.966507 (+ 340us) server_negotiation.cc:300] Negotiation successful
0202 15:04:21.966662 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:04:21.967689 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.951004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.2.164.65:40421)
0202 15:04:21.951422 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.951464 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.952481 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.953045 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.953057 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.953888 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:21.954302 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.954314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.955351 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.955357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.956553 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.956563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.957148 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.957823 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:21.957864 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:21.960073 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:21.964525 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:21.964528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:21.964530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:21.964735 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:21.965046 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:21.965049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:21.965051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:21.965100 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:21.965828 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:21.965834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:21.965913 (+ 79us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.966888 (+ 975us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.967265 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":73,"spinlock_wait_cycles":4864,"thread_start_us":118,"threads_started":1}
I20260202 15:04:21.967711 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.951004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:47623)
0202 15:04:21.951422 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.951464 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.952502 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.953054 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.953058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.953654 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:21.954369 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.954378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.955385 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.955389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.956553 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.956563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.957291 (+ 728us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.958014 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:21.958032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:21.960571 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:21.965186 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:21.965191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:21.965196 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:21.965384 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:21.965679 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:21.965683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:21.965685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:21.965738 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:21.966227 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:21.966232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:21.966311 (+ 79us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.967006 (+ 695us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.967171 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":20,"spinlock_wait_cycles":26624,"thread_start_us":143,"threads_started":1}
I20260202 15:04:21.967707 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:21.951962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.2.164.65:34831)
0202 15:04:21.952363 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:21.952376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:21.952481 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:21.953275 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:21.953278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:21.953654 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:21.954369 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.954378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.955189 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:21.955192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:21.957209 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:21.957223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:21.958292 (+ 1069us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:21.958848 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:21.958869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:21.961158 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:21.963339 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:21.963345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:21.963358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:21.964215 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:21.964529 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:21.964531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:21.964533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:21.964697 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:21.965248 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:21.965255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:21.965588 (+ 333us) client_negotiation.cc:769] Sending connection context
0202 15:04:21.966455 (+ 867us) client_negotiation.cc:240] Negotiation successful
0202 15:04:21.966799 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"spinlock_wait_cycles":69632,"thread_start_us":92,"threads_started":1}
I20260202 15:04:21.969060 10440 heartbeater.cc:344] Connected to a master server at 127.2.164.126:45883
I20260202 15:04:21.969229 10442 heartbeater.cc:344] Connected to a master server at 127.2.164.125:37549
I20260202 15:04:21.969319 10440 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:21.969319 10442 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:21.969949 10440 heartbeater.cc:507] Master 127.2.164.126:45883 requested a full tablet report, sending...
I20260202 15:04:21.969949 10442 heartbeater.cc:507] Master 127.2.164.125:37549 requested a full tablet report, sending...
I20260202 15:04:21.970554 10441 heartbeater.cc:344] Connected to a master server at 127.2.164.124:41495
I20260202 15:04:21.970647 10441 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:21.970865 10441 heartbeater.cc:507] Master 127.2.164.124:41495 requested a full tablet report, sending...
I20260202 15:04:21.971464 9449 ts_manager.cc:194] Re-registered known tserver with Master: 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971)
I20260202 15:04:21.971695 10057 ts_manager.cc:194] Registered new tserver with Master: 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971)
I20260202 15:04:21.971927 10248 ts_manager.cc:194] Registered new tserver with Master: 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971)
I20260202 15:04:21.972223 9449 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:47623
W20260202 15:04:22.077926 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:22.078186 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:22.078277 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:22.081722 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:22.081845 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:22.081971 10447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:22.086261 10447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:32967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=42519
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:22.087409 10447 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:22.088279 10447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:22.094854 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:22.094870 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:22.095028 10447 server_base.cc:1060] running on GCE node
W20260202 15:04:22.094831 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:22.095551 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:22.096083 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:22.097271 10447 hybrid_clock.cc:648] HybridClock initialized: now 1770044662097253 us; error 41 us; skew 500 ppm
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:22.099896 10447 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260202 15:04:22.100942 10447 webserver.cc:492] Webserver started at http://127.2.164.66:42519/ using document root <none> and password file <none>
I20260202 15:04:22.101516 10447 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:22.101605 10447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:22.104903 10447 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:22.106719 10462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:22.107785 10447 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260202 15:04:22.107915 10447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "aca6cb50f08848fcad0733a5a39d4282"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "cae981eec9a365e45548bf11a96d57f9"
server_key_iv: "283e035dd248e910fb1f9ac15b64c15b"
server_key_version: "encryptionkey@0"
I20260202 15:04:22.108336 10447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:22.121965 10447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:22.124862 10447 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:22.125115 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:22.125664 10447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:22.126562 10447 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:22.126633 10447 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:22.126699 10447 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:22.126749 10447 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:22.135728 10447 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:32967
I20260202 15:04:22.135743 10575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:32967 every 8 connection(s)
I20260202 15:04:22.136708 10447 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:22.142974 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 10447
I20260202 15:04:22.143152 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9788
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260202 15:04:22.151862 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.139761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:39051 (local address 127.2.164.126:45883)
0202 15:04:22.139924 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.139930 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.140179 (+ 249us) server_negotiation.cc:408] Connection header received
0202 15:04:22.140565 (+ 386us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.140568 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.140645 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.140726 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.142251 (+ 1525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.142811 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.143808 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.144058 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.146977 (+ 2919us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.146997 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.147000 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.147036 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.148842 (+ 1806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.149488 (+ 646us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.149492 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.149494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.149548 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.149830 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.149833 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.149835 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.150041 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.150166 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.151540 (+ 1374us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.151676 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260202 15:04:22.152390 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:45485
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:45883
--webserver_port=38555
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495 with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260202 15:04:22.152820 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.138409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:58319 (local address 127.2.164.124:41495)
0202 15:04:22.138546 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.138550 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.139943 (+ 1393us) server_negotiation.cc:408] Connection header received
0202 15:04:22.140460 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.140464 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.140528 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.140607 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.142113 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.142901 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.144146 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.144339 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.148235 (+ 3896us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.148257 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.148264 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.148301 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.151017 (+ 2716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151605 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.151608 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.151610 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.151661 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151939 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.151944 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.151947 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.152142 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.152217 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.152572 (+ 355us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.152684 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260202 15:04:22.153039 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.138518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:41873 (local address 127.2.164.125:37549)
0202 15:04:22.138639 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.138643 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.139345 (+ 702us) server_negotiation.cc:408] Connection header received
0202 15:04:22.140846 (+ 1501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.140850 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.140907 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.141014 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.142144 (+ 1130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.142901 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.144469 (+ 1568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.144696 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.148589 (+ 3893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.148613 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.148619 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.148671 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.150834 (+ 2163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151643 (+ 809us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.151649 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.151654 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.151710 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.152009 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.152012 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.152014 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.152202 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.152274 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.152627 (+ 353us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.152769 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:04:22.153518 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.139581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:39051)
0202 15:04:22.140089 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.140101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.140241 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.140795 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.140803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.141824 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:22.142069 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.142078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.142977 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.142980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.143664 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.143678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.144275 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.144902 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:22.144940 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:22.146811 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:22.149042 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.149048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.149061 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.149396 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.149626 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.149629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.149632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.149729 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.150937 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:22.150942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:22.151220 (+ 278us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.152397 (+ 1177us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.152684 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":71,"threads_started":1}
I20260202 15:04:22.153532 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.138724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.2.164.66:58319)
0202 15:04:22.139836 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.139849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.140253 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.140805 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.140809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.141371 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:22.141932 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.141943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.143049 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.143054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.143993 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.144002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.144463 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.145629 (+ 1166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:22.145650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:22.148088 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:22.151134 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.151139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.151498 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.151771 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.151777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.151822 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.152286 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:22.152289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:22.152370 (+ 81us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.152954 (+ 584us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.153106 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":20,"spinlock_wait_cycles":8448,"thread_start_us":92,"threads_started":1}
I20260202 15:04:22.153730 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.138724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.2.164.66:41873)
0202 15:04:22.139185 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.139227 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.140238 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.141060 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.141063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.141385 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:22.141966 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.141975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.143039 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.143043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.144335 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.144343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.144809 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.145980 (+ 1171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:22.146002 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:22.148460 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:22.151182 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.151192 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.151531 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.151830 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.151836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.151840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.151901 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.152347 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:22.152349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:22.152431 (+ 82us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.153284 (+ 853us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.153430 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"spinlock_wait_cycles":13440,"thread_start_us":122,"threads_started":1}
I20260202 15:04:22.154819 10576 heartbeater.cc:344] Connected to a master server at 127.2.164.126:45883
I20260202 15:04:22.154870 10578 heartbeater.cc:344] Connected to a master server at 127.2.164.125:37549
I20260202 15:04:22.155074 10576 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:22.155074 10578 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:22.155743 10576 heartbeater.cc:507] Master 127.2.164.126:45883 requested a full tablet report, sending...
I20260202 15:04:22.155743 10578 heartbeater.cc:507] Master 127.2.164.125:37549 requested a full tablet report, sending...
I20260202 15:04:22.156561 10577 heartbeater.cc:344] Connected to a master server at 127.2.164.124:41495
I20260202 15:04:22.156680 10577 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:22.156834 9449 ts_manager.cc:194] Re-registered known tserver with Master: aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967)
I20260202 15:04:22.156929 10577 heartbeater.cc:507] Master 127.2.164.124:41495 requested a full tablet report, sending...
I20260202 15:04:22.157042 10057 ts_manager.cc:194] Registered new tserver with Master: aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967)
I20260202 15:04:22.157526 9449 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:39051
I20260202 15:04:22.157647 10248 ts_manager.cc:194] Registered new tserver with Master: aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967)
W20260202 15:04:22.265565 10583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:22.265815 10583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:22.265945 10583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:22.269462 10583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:22.269527 10583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:22.269651 10583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:22.274080 10583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=38555
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:45883,127.2.164.125:37549,127.2.164.124:41495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:22.275167 10583 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:22.276028 10583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:22.282760 10591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:22.282946 10583 server_base.cc:1060] running on GCE node
W20260202 15:04:22.282760 10589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:22.282783 10588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:22.283435 10583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:22.283983 10583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:22.285254 10583 hybrid_clock.cc:648] HybridClock initialized: now 1770044662285211 us; error 39 us; skew 500 ppm
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:22.288405 10583 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260202 15:04:22.289603 10583 webserver.cc:492] Webserver started at http://127.2.164.67:38555/ using document root <none> and password file <none>
I20260202 15:04:22.290206 10583 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:22.290298 10583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:22.293857 10583 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:22.295717 10598 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:22.296716 10583 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:22.296849 10583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8"
format_stamp: "Formatted at 2026-02-02 15:04:19 on dist-test-slave-6zzp"
server_key: "a30432df9d09541a9eee2a16cbef2c0f"
server_key_iv: "68dccfe2c7bc52902a67bc6a66c77573"
server_key_version: "encryptionkey@0"
I20260202 15:04:22.297320 10583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:22.351416 10583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:22.354395 10583 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:22.354597 10583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:22.355163 10583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:22.356060 10583 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:22.356132 10583 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:22.356201 10583 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:22.356253 10583 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:22.367179 10583 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45485
I20260202 15:04:22.367234 10711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:45485 every 8 connection(s)
I20260202 15:04:22.368161 10583 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:22.371312 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 10583
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
Feb 02 15:04:22 dist-test-slave-6zzp krb5kdc[9404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260202 15:04:22.385725 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.374014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.2.164.124:41495)
0202 15:04:22.376233 (+ 2219us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.376238 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.376249 (+ 11us) server_negotiation.cc:408] Connection header received
0202 15:04:22.376286 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.376289 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.376337 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.376434 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.377844 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.378594 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.379667 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.379885 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.381425 (+ 1540us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.381444 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.381450 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.381482 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.383713 (+ 2231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.384187 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.384191 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.384193 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.384242 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.384500 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.384503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.384504 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.384682 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.384833 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.385413 (+ 580us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.385534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":73,"threads_started":1}
I20260202 15:04:22.386672 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.369928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:46447 (local address 127.2.164.124:41495)
0202 15:04:22.370055 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.370059 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.370740 (+ 681us) server_negotiation.cc:408] Connection header received
0202 15:04:22.371882 (+ 1142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.371886 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.371939 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.371992 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.373922 (+ 1930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.374695 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.375689 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.375894 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.380744 (+ 4850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.380763 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.380770 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.380802 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.385125 (+ 4323us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.385562 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.385566 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.385568 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.385614 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.385877 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.385880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.385882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.386039 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.386118 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.386451 (+ 333us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.386553 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260202 15:04:22.386893 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.371100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:40599)
0202 15:04:22.371668 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.371682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.371818 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.372385 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.372394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.373035 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:22.373743 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.373759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.374789 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.374795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.375371 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.375382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.375577 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.376246 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:22.376280 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:22.379443 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:22.383672 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.383679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.383694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.384074 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.384511 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.384515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.384517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.384636 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.385128 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:22.385136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:22.385467 (+ 331us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.385707 (+ 240us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.386021 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"spinlock_wait_cycles":7936,"thread_start_us":76,"threads_started":1}
I20260202 15:04:22.386880 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.370161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41495 (local address 127.2.164.67:46447)
0202 15:04:22.370604 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.370647 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.371636 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.372383 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.372394 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.373050 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:22.373741 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.373757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.374856 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.374860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.375537 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.375548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.376019 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.377106 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:22.377125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:22.380639 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:22.385239 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.385242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.385244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.385455 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.385723 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.385726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.385728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.385775 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.386190 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:22.386194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:22.386269 (+ 75us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.386371 (+ 102us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.386499 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"spinlock_wait_cycles":46848,"thread_start_us":89,"threads_started":1}
I20260202 15:04:22.387372 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.371351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40599 (local address 127.2.164.126:45883)
0202 15:04:22.371497 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.371502 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.371744 (+ 242us) server_negotiation.cc:408] Connection header received
0202 15:04:22.371933 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.371937 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.371985 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.372035 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.373908 (+ 1873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.374651 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.375878 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.376079 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.379623 (+ 3544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.379642 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.379645 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.379671 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.383248 (+ 3577us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.384356 (+ 1108us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.384360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.384362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.384406 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.384752 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.384756 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.384758 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.384941 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.385066 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.386981 (+ 1915us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.387100 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:04:22.387574 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.374673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.2.164.126:45883)
0202 15:04:22.376642 (+ 1969us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.376646 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.376986 (+ 340us) server_negotiation.cc:408] Connection header received
0202 15:04:22.377049 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.377052 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.377109 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.377190 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.378301 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.379076 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.381403 (+ 2327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.381593 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.382662 (+ 1069us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.382682 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.382686 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.382721 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.384738 (+ 2017us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.385226 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.385229 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.385231 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.385419 (+ 188us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.385690 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.385693 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.385695 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.385881 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.385996 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.387283 (+ 1287us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.387410 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1898,"thread_start_us":69,"threads_started":1}
I20260202 15:04:22.388789 10713 heartbeater.cc:344] Connected to a master server at 127.2.164.124:41495
I20260202 15:04:22.389106 10713 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:22.389744 10713 heartbeater.cc:507] Master 127.2.164.124:41495 requested a full tablet report, sending...
I20260202 15:04:22.390354 10712 heartbeater.cc:344] Connected to a master server at 127.2.164.126:45883
I20260202 15:04:22.390424 10712 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:22.390456 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.369875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40613 (local address 127.2.164.125:37549)
0202 15:04:22.370027 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.370031 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.370736 (+ 705us) server_negotiation.cc:408] Connection header received
0202 15:04:22.371874 (+ 1138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.371878 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.371937 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.371992 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.373911 (+ 1919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.374695 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.375689 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.376505 (+ 816us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.380270 (+ 3765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.380287 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.380291 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.380314 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.384243 (+ 3929us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.389227 (+ 4984us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.389231 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.389233 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.389295 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.389571 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.389575 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.389577 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.389747 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.389838 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.390175 (+ 337us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.390296 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260202 15:04:22.390617 10712 heartbeater.cc:507] Master 127.2.164.126:45883 requested a full tablet report, sending...
I20260202 15:04:22.390904 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.370161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:37549 (local address 127.2.164.67:40613)
0202 15:04:22.370604 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.370647 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.371636 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.372383 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.372394 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.373388 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:22.373741 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.373757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.374856 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.374860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.375537 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.375548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.376627 (+ 1079us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.377133 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:22.377145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:22.380151 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:22.388296 (+ 8145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.388300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.388302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.388501 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.389414 (+ 913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:22.389417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:22.389419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:22.389472 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:22.389893 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:22.389896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:22.389967 (+ 71us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.390578 (+ 611us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.390725 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":23,"spinlock_wait_cycles":4352,"thread_start_us":133,"threads_started":1}
I20260202 15:04:22.391597 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.375246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.2.164.125:37549)
0202 15:04:22.375767 (+ 521us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.375774 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.378596 (+ 2822us) server_negotiation.cc:408] Connection header received
0202 15:04:22.378642 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.378647 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.378695 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.378773 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.380043 (+ 1270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.380790 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.381741 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.381969 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.383146 (+ 1177us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.383168 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.383176 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.383209 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.390067 (+ 6858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.390510 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.390514 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.390515 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.390560 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.390818 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.390821 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.390823 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.390970 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.391091 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.391335 (+ 244us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.391450 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":61,"threads_started":1}
I20260202 15:04:22.391855 10248 ts_manager.cc:194] Registered new tserver with Master: 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485)
I20260202 15:04:22.390748 9449 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260202 15:04:22.393253 9451 ts_manager.cc:194] Re-registered known tserver with Master: 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485)
I20260202 15:04:22.393482 10714 heartbeater.cc:344] Connected to a master server at 127.2.164.125:37549
I20260202 15:04:22.393555 10714 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:22.393730 10714 heartbeater.cc:507] Master 127.2.164.125:37549 requested a full tablet report, sending...
W20260202 15:04:22.393857 9449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:22.393970 9451 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:40599
I20260202 15:04:22.394500 10057 ts_manager.cc:194] Registered new tserver with Master: 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485)
I20260202 15:04:22.410252 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.405839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38971 (local address 127.0.0.1:60000)
0202 15:04:22.406118 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.406138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.406243 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.407471 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.407475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.407495 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:22.407733 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.407740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.408737 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.408743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.409699 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.409707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.409902 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.409932 (+ 30us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.410014 (+ 82us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.410084 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":112,"threads_started":1}
I20260202 15:04:22.410708 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.406570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32967 (local address 127.0.0.1:57238)
0202 15:04:22.406818 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.406834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.406921 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.407731 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.407735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.407754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:22.407980 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.407986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.408886 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.408891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.409922 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.409930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.410201 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.410224 (+ 23us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.410287 (+ 63us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.410587 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260202 15:04:22.411010 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.406978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.2.164.66:32967)
0202 15:04:22.407288 (+ 310us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.407294 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.407318 (+ 24us) server_negotiation.cc:408] Connection header received
0202 15:04:22.407400 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.407404 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.407575 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.407742 (+ 167us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:22.408103 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.408765 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.410055 (+ 1290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.410575 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.410694 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.410763 (+ 69us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.410867 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260202 15:04:22.412110 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.406024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.2.164.65:38971)
0202 15:04:22.406382 (+ 358us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.406389 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.406414 (+ 25us) server_negotiation.cc:408] Connection header received
0202 15:04:22.406508 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.406518 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.407305 (+ 787us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.407850 (+ 545us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:22.407955 (+ 105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.408621 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.410459 (+ 1838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.411018 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.411112 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.411863 (+ 751us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.411976 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260202 15:04:22.413236 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.406593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45485 (local address 127.0.0.1:36560)
0202 15:04:22.409168 (+ 2575us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.409182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.409382 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.410474 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.410477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.410495 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:22.410716 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.410723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.411987 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.411991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.412823 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.412832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.412967 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.412984 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.413053 (+ 69us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.413122 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2372,"thread_start_us":91,"threads_started":1}
I20260202 15:04:22.413916 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.406706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.2.164.67:45485)
0202 15:04:22.408902 (+ 2196us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.408908 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.409320 (+ 412us) server_negotiation.cc:408] Connection header received
0202 15:04:22.409524 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.409529 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.409667 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.410214 (+ 547us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:22.410849 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.411857 (+ 1008us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.413107 (+ 1250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.413616 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.413691 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.413741 (+ 50us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.413800 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2102,"thread_start_us":83,"threads_started":1}
I20260202 15:04:22.414211 10510 tablet_service.cc:1505] Processing CreateTablet for tablet 0af772e79ef3471f9284381db5868520 (DEFAULT_TABLE table=test-table [id=f618f7edeb524c4eb2725941c293fe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:22.414724 10374 tablet_service.cc:1505] Processing CreateTablet for tablet 0af772e79ef3471f9284381db5868520 (DEFAULT_TABLE table=test-table [id=f618f7edeb524c4eb2725941c293fe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:22.415256 10510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af772e79ef3471f9284381db5868520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:22.415561 10374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af772e79ef3471f9284381db5868520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:22.415850 10646 tablet_service.cc:1505] Processing CreateTablet for tablet 0af772e79ef3471f9284381db5868520 (DEFAULT_TABLE table=test-table [id=f618f7edeb524c4eb2725941c293fe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:22.416601 10646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af772e79ef3471f9284381db5868520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:22.422071 10743 tablet_bootstrap.cc:492] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Bootstrap starting.
I20260202 15:04:22.423040 10744 tablet_bootstrap.cc:492] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Bootstrap starting.
I20260202 15:04:22.423843 10745 tablet_bootstrap.cc:492] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Bootstrap starting.
I20260202 15:04:22.424628 10743 tablet_bootstrap.cc:654] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:22.425132 10744 tablet_bootstrap.cc:654] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:22.425940 10744 log.cc:826] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:22.426049 10743 log.cc:826] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:22.426204 10745 tablet_bootstrap.cc:654] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:22.426960 10745 log.cc:826] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:22.427889 10744 tablet_bootstrap.cc:492] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: No bootstrap required, opened a new log
I20260202 15:04:22.427889 10743 tablet_bootstrap.cc:492] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: No bootstrap required, opened a new log
I20260202 15:04:22.428086 10743 ts_tablet_manager.cc:1403] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:04:22.428086 10744 ts_tablet_manager.cc:1403] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260202 15:04:22.428561 10745 tablet_bootstrap.cc:492] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: No bootstrap required, opened a new log
I20260202 15:04:22.428763 10745 ts_tablet_manager.cc:1403] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260202 15:04:22.431187 10744 raft_consensus.cc:359] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.431185 10743 raft_consensus.cc:359] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.431181 10745 raft_consensus.cc:359] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.431404 10743 raft_consensus.cc:385] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:22.431404 10745 raft_consensus.cc:385] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:22.431464 10745 raft_consensus.cc:740] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604b79fe893d4460ad4c782af7e87396, State: Initialized, Role: FOLLOWER
I20260202 15:04:22.431480 10743 raft_consensus.cc:740] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea2fcb4eaa94d1086db8301f7f2b3c8, State: Initialized, Role: FOLLOWER
I20260202 15:04:22.431545 10744 raft_consensus.cc:385] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:22.431612 10744 raft_consensus.cc:740] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca6cb50f08848fcad0733a5a39d4282, State: Initialized, Role: FOLLOWER
I20260202 15:04:22.431973 10745 consensus_queue.cc:260] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.431985 10743 consensus_queue.cc:260] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.431985 10744 consensus_queue.cc:260] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.432873 10744 ts_tablet_manager.cc:1434] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260202 15:04:22.432873 10743 ts_tablet_manager.cc:1434] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:04:22.432881 10745 ts_tablet_manager.cc:1434] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260202 15:04:22.433125 10440 heartbeater.cc:499] Master 127.2.164.126:45883 was elected leader, sending a full tablet report...
I20260202 15:04:22.433478 10712 heartbeater.cc:499] Master 127.2.164.126:45883 was elected leader, sending a full tablet report...
I20260202 15:04:22.434438 10576 heartbeater.cc:499] Master 127.2.164.126:45883 was elected leader, sending a full tablet report...
W20260202 15:04:22.450915 10443 tablet.cc:2378] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:04:22.497241 10750 raft_consensus.cc:493] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:22.497462 10750 raft_consensus.cc:515] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.498860 10750 leader_election.cc:290] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971), aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967)
I20260202 15:04:22.501802 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.499049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38971 (local address 127.2.164.67:43917)
0202 15:04:22.499262 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.499280 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.499382 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.499657 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.499659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.499685 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:22.499904 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.499910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.500618 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.500624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.501296 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.501310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.501464 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.501486 (+ 22us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.501541 (+ 55us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.501600 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"spinlock_wait_cycles":1792}
I20260202 15:04:22.501811 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.499132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32967 (local address 127.2.164.67:33555)
0202 15:04:22.499316 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:22.499332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:22.499420 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:22.499660 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:22.499663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:22.499694 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:22.499904 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.499910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.500648 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.500654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:22.501393 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:22.501402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.501522 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.501546 (+ 24us) client_negotiation.cc:769] Sending connection context
0202 15:04:22.501599 (+ 53us) client_negotiation.cc:240] Negotiation successful
0202 15:04:22.501656 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"spinlock_wait_cycles":5632}
I20260202 15:04:22.502187 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.499171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:43917 (local address 127.2.164.65:38971)
0202 15:04:22.499359 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.499364 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.499380 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:04:22.499508 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.499510 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.499554 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.499636 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:22.500019 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.500508 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.501427 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.501908 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.501949 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.502008 (+ 59us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.502073 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260202 15:04:22.502403 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.499296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:33555 (local address 127.2.164.66:32967)
0202 15:04:22.499419 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.499424 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.499440 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:04:22.499502 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.499506 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.499555 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.499638 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:22.500026 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.500525 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.501584 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.502123 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.502168 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.502224 (+ 56us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.502297 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260202 15:04:22.502789 10394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0af772e79ef3471f9284381db5868520" candidate_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604b79fe893d4460ad4c782af7e87396" is_pre_election: true
I20260202 15:04:22.502997 10530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0af772e79ef3471f9284381db5868520" candidate_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca6cb50f08848fcad0733a5a39d4282" is_pre_election: true
I20260202 15:04:22.503095 10394 raft_consensus.cc:2468] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea2fcb4eaa94d1086db8301f7f2b3c8 in term 0.
I20260202 15:04:22.503213 10530 raft_consensus.cc:2468] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea2fcb4eaa94d1086db8301f7f2b3c8 in term 0.
I20260202 15:04:22.503618 10600 leader_election.cc:304] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea2fcb4eaa94d1086db8301f7f2b3c8, 604b79fe893d4460ad4c782af7e87396; no voters:
I20260202 15:04:22.503885 10750 raft_consensus.cc:2804] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:22.503963 10750 raft_consensus.cc:493] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:22.504027 10750 raft_consensus.cc:3060] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:22.505606 10750 raft_consensus.cc:515] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.506027 10750 leader_election.cc:290] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971), aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967)
I20260202 15:04:22.506369 10394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0af772e79ef3471f9284381db5868520" candidate_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604b79fe893d4460ad4c782af7e87396"
I20260202 15:04:22.506422 10530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0af772e79ef3471f9284381db5868520" candidate_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca6cb50f08848fcad0733a5a39d4282"
I20260202 15:04:22.506487 10394 raft_consensus.cc:3060] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:22.506574 10530 raft_consensus.cc:3060] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:22.507436 10394 raft_consensus.cc:2468] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea2fcb4eaa94d1086db8301f7f2b3c8 in term 1.
I20260202 15:04:22.507576 10530 raft_consensus.cc:2468] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea2fcb4eaa94d1086db8301f7f2b3c8 in term 1.
I20260202 15:04:22.507790 10600 leader_election.cc:304] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea2fcb4eaa94d1086db8301f7f2b3c8, 604b79fe893d4460ad4c782af7e87396; no voters:
I20260202 15:04:22.507987 10750 raft_consensus.cc:2804] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:22.508213 10750 raft_consensus.cc:697] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 LEADER]: Becoming Leader. State: Replica: 2ea2fcb4eaa94d1086db8301f7f2b3c8, State: Running, Role: LEADER
I20260202 15:04:22.508545 10750 consensus_queue.cc:237] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } }
I20260202 15:04:22.512167 9450 catalog_manager.cc:5654] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } health_report { overall_health: UNKNOWN } } }
I20260202 15:04:22.537976 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.534802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.2.164.67:45485)
0202 15:04:22.534969 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.534973 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.534991 (+ 18us) server_negotiation.cc:408] Connection header received
0202 15:04:22.535091 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.535094 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.535148 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.535256 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:22.535588 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.536088 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.536751 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.536957 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.537072 (+ 115us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:22.537529 (+ 457us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:22.537682 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.537763 (+ 81us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.537815 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:04:22.544250 10530 raft_consensus.cc:1275] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Refusing update from remote peer 2ea2fcb4eaa94d1086db8301f7f2b3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:22.544250 10394 raft_consensus.cc:1275] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 1 FOLLOWER]: Refusing update from remote peer 2ea2fcb4eaa94d1086db8301f7f2b3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:22.544950 10750 consensus_queue.cc:1048] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:22.545164 10752 consensus_queue.cc:1048] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:22.553833 10755 mvcc.cc:204] Tried to move back new op lower bound from 7250102937773531136 to 7250102937638023168. Current Snapshot: MvccSnapshot[applied={T|T < 7250102937773531136}]
I20260202 15:04:22.556913 10758 mvcc.cc:204] Tried to move back new op lower bound from 7250102937773531136 to 7250102937638023168. Current Snapshot: MvccSnapshot[applied={T|T < 7250102937773531136}]
I20260202 15:04:22.563958 9450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:22.564127 9450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:22.568707 9450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 94296279378342b0afd24fa731eebc9f: Sending DeleteTablet for 3 replicas of tablet 0af772e79ef3471f9284381db5868520
I20260202 15:04:22.569417 10374 tablet_service.cc:1552] Processing DeleteTablet for tablet 0af772e79ef3471f9284381db5868520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:22 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260202 15:04:22.569765 10510 tablet_service.cc:1552] Processing DeleteTablet for tablet 0af772e79ef3471f9284381db5868520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:22 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57238
I20260202 15:04:22.569927 10766 tablet_replica.cc:333] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: stopping tablet replica
I20260202 15:04:22.570211 10766 raft_consensus.cc:2243] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:22.570251 10767 tablet_replica.cc:333] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: stopping tablet replica
I20260202 15:04:22.570472 10767 raft_consensus.cc:2243] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:22.570670 10766 raft_consensus.cc:2272] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:22.570657 10646 tablet_service.cc:1552] Processing DeleteTablet for tablet 0af772e79ef3471f9284381db5868520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:22 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:36560
I20260202 15:04:22.570791 10767 raft_consensus.cc:2272] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:22.572242 10767 ts_tablet_manager.cc:1916] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:22.572249 10766 ts_tablet_manager.cc:1916] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:22.574765 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9420
I20260202 15:04:22.575490 10767 ts_tablet_manager.cc:1929] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:22.575605 10767 log.cc:1199] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/0af772e79ef3471f9284381db5868520
I20260202 15:04:22.575970 10767 ts_tablet_manager.cc:1950] T 0af772e79ef3471f9284381db5868520 P aca6cb50f08848fcad0733a5a39d4282: Deleting consensus metadata
I20260202 15:04:22.576027 10766 ts_tablet_manager.cc:1929] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:22.576135 10766 log.cc:1199] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/0af772e79ef3471f9284381db5868520
I20260202 15:04:22.576479 10766 ts_tablet_manager.cc:1950] T 0af772e79ef3471f9284381db5868520 P 604b79fe893d4460ad4c782af7e87396: Deleting consensus metadata
I20260202 15:04:22.581310 10768 tablet_replica.cc:333] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: stopping tablet replica
I20260202 15:04:22.581665 10768 raft_consensus.cc:2243] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:22.582098 10768 raft_consensus.cc:2272] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260202 15:04:22.583876 10330 connection.cc:537] server connection from 127.0.0.1:60000 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260202 15:04:22.584122 10768 ts_tablet_manager.cc:1916] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260202 15:04:22.584334 10463 connection.cc:537] server connection from 127.0.0.1:57238 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260202 15:04:22.587093 10768 ts_tablet_manager.cc:1929] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:22.587208 10768 log.cc:1199] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/0af772e79ef3471f9284381db5868520
I20260202 15:04:22.587556 10768 ts_tablet_manager.cc:1950] T 0af772e79ef3471f9284381db5868520 P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Deleting consensus metadata
W20260202 15:04:22.588732 10601 connection.cc:441] server connection from 127.0.0.1:36560 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36560 (request call id 1) could send its response
I20260202 15:04:22.595435 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.586507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.2.164.125:37549)
0202 15:04:22.586634 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.586638 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.587166 (+ 528us) server_negotiation.cc:408] Connection header received
0202 15:04:22.587467 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.587474 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.587537 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.587598 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.589135 (+ 1537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.589825 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.590487 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.590704 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.591954 (+ 1250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.591972 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.591974 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.591997 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.593764 (+ 1767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.594236 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.594240 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.594241 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.594284 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.594572 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.594577 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.594579 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.594767 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.594905 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.595136 (+ 231us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.595264 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260202 15:04:22.595636 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:22.585759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.2.164.124:41495)
0202 15:04:22.585900 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:22.585904 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:22.586466 (+ 562us) server_negotiation.cc:408] Connection header received
0202 15:04:22.586782 (+ 316us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:22.586786 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:22.586852 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:22.586904 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:22.588745 (+ 1841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.589651 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:22.590541 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:22.590746 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:22.592286 (+ 1540us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:22.592310 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:22.592316 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:22.592353 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:22.594056 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.594473 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.594479 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.594484 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.594538 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:22.594791 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:22.594794 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:22.594796 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:22.594995 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:22.595119 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:22.595417 (+ 298us) server_negotiation.cc:300] Negotiation successful
0202 15:04:22.595527 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:04:23.560232 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:23.559693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:53357)
0202 15:04:23.560005 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:23.560110 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
W20260202 15:04:23.560562 10712 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:45883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:23.563602 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:23.563098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:34999)
0202 15:04:23.563396 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:23.563480 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260202 15:04:23.564092 10440 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:45883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:23.565032 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:23.564396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:53295)
0202 15:04:23.564691 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:23.564765 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":72,"threads_started":1}
W20260202 15:04:23.565270 10576 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:45883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:24.068696 10779 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94296279378342b0afd24fa731eebc9f)
I20260202 15:04:24.068840 10779 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } }
I20260202 15:04:24.069392 10779 leader_election.cc:290] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883), f11e9461627f442096e6fdcba2f270b7 (127.2.164.125:37549)
I20260202 15:04:24.070017 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.069603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59686)
0202 15:04:24.069821 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.069909 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260202 15:04:24.070175 10781 raft_consensus.cc:493] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94296279378342b0afd24fa731eebc9f)
W20260202 15:04:24.070292 10232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:24.070389 10074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aea978ffa0f48749984cdc07f255126" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f11e9461627f442096e6fdcba2f270b7" is_pre_election: true
I20260202 15:04:24.070349 10781 raft_consensus.cc:515] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } }
I20260202 15:04:24.070842 10074 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aea978ffa0f48749984cdc07f255126 in term 1.
I20260202 15:04:24.071204 10232 leader_election.cc:304] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aea978ffa0f48749984cdc07f255126, f11e9461627f442096e6fdcba2f270b7; no voters:
I20260202 15:04:24.071224 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.070934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59702)
0202 15:04:24.071092 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.071158 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260202 15:04:24.071386 10232 leader_election.cc:336] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883): Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:24.071408 10779 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260202 15:04:24.071506 10779 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 94296279378342b0afd24fa731eebc9f)
I20260202 15:04:24.071552 10779 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:04:24.071677 10781 leader_election.cc:290] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883), 7aea978ffa0f48749984cdc07f255126 (127.2.164.124:41495)
I20260202 15:04:24.072110 10258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f11e9461627f442096e6fdcba2f270b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7aea978ffa0f48749984cdc07f255126" is_pre_election: true
I20260202 15:04:24.072229 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.071767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59716)
0202 15:04:24.072043 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.072103 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
W20260202 15:04:24.072352 10048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:24.072860 10779 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } }
I20260202 15:04:24.073129 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.072880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59720)
0202 15:04:24.073020 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.073068 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260202 15:04:24.073128 10258 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f11e9461627f442096e6fdcba2f270b7 in current term 2: Already voted for candidate 7aea978ffa0f48749984cdc07f255126 in this term.
I20260202 15:04:24.073283 10779 leader_election.cc:290] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 2 election: Requested vote from peers 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883), f11e9461627f442096e6fdcba2f270b7 (127.2.164.125:37549)
W20260202 15:04:24.073519 10048 leader_election.cc:336] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883): Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:24.073582 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.073362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59728)
0202 15:04:24.073470 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.073521 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260202 15:04:24.073657 10050 leader_election.cc:304] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f11e9461627f442096e6fdcba2f270b7; no voters: 7aea978ffa0f48749984cdc07f255126, 94296279378342b0afd24fa731eebc9f
I20260202 15:04:24.073876 10781 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:04:24.074211 10074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7aea978ffa0f48749984cdc07f255126" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f11e9461627f442096e6fdcba2f270b7"
I20260202 15:04:24.074270 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.073991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59730)
0202 15:04:24.074152 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.074208 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260202 15:04:24.074378 10232 leader_election.cc:336] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883): Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
I20260202 15:04:24.074949 10781 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260202 15:04:24.075812 10074 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aea978ffa0f48749984cdc07f255126 in term 2.
I20260202 15:04:24.076133 10232 leader_election.cc:304] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7aea978ffa0f48749984cdc07f255126, f11e9461627f442096e6fdcba2f270b7; no voters: 94296279378342b0afd24fa731eebc9f
I20260202 15:04:24.076308 10779 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:04:24.076488 10779 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [term 2 LEADER]: Becoming Leader. State: Replica: 7aea978ffa0f48749984cdc07f255126, State: Running, Role: LEADER
I20260202 15:04:24.076732 10779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } }
I20260202 15:04:24.077875 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aea978ffa0f48749984cdc07f255126. Latest consensus state: current_term: 2 leader_uuid: "7aea978ffa0f48749984cdc07f255126" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:24.077994 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:24.078277 10787 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:24.080408 10787 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:24.080832 10787 catalog_manager.cc:1269] Loaded cluster ID: dbfff5e6ac374cf4b4b93b06201d8cd4
I20260202 15:04:24.080885 10787 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:24.081483 10787 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:24.081907 10787 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Loaded TSK: 0
I20260202 15:04:24.082629 10787 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:04:24.118772 10246 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:04:24.121248 10246 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:24.124043 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.123780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59744)
0202 15:04:24.123907 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.123977 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260202 15:04:24.124025 10074 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [term 2 FOLLOWER]: Refusing update from remote peer 7aea978ffa0f48749984cdc07f255126: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260202 15:04:24.124531 10785 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260202 15:04:24.124817 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.124592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.0.0.1:59748)
0202 15:04:24.124705 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.124754 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260202 15:04:24.125080 10232 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 -> Peer 94296279378342b0afd24fa731eebc9f (127.2.164.126:45883): Couldn't send request to peer 94296279378342b0afd24fa731eebc9f. Status: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260202 15:04:24.126003 10781 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aea978ffa0f48749984cdc07f255126. Latest consensus state: current_term: 2 leader_uuid: "7aea978ffa0f48749984cdc07f255126" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:24.126107 10781 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:24.127236 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7aea978ffa0f48749984cdc07f255126" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:24.127362 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:24.127808 10781 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7aea978ffa0f48749984cdc07f255126" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94296279378342b0afd24fa731eebc9f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 45883 } } peers { permanent_uuid: "f11e9461627f442096e6fdcba2f270b7" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 37549 } attrs { promote: false } } peers { permanent_uuid: "7aea978ffa0f48749984cdc07f255126" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41495 } attrs { promote: false } } }
I20260202 15:04:24.127930 10781 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11e9461627f442096e6fdcba2f270b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260202 15:04:24.136932 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.133242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32967 (local address 127.0.0.1:57248)
0202 15:04:24.133885 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.133900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:24.134069 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:24.134411 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:24.134415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:24.134432 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:24.134723 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.134733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.135667 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.135671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:24.136558 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.136567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.136664 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.136685 (+ 21us) client_negotiation.cc:769] Sending connection context
0202 15:04:24.136735 (+ 50us) client_negotiation.cc:240] Negotiation successful
0202 15:04:24.136798 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542}
I20260202 15:04:24.138077 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.133341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.2.164.66:32967)
0202 15:04:24.133611 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:24.133616 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:24.134051 (+ 435us) server_negotiation.cc:408] Connection header received
0202 15:04:24.134206 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:24.134209 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:24.134279 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:24.134381 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:24.134857 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.135532 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.137288 (+ 1756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.137786 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.137833 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:24.137884 (+ 51us) server_negotiation.cc:300] Negotiation successful
0202 15:04:24.137943 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260202 15:04:24.138088 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.133961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.2.164.65:38971)
0202 15:04:24.134511 (+ 550us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:24.134516 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:24.134536 (+ 20us) server_negotiation.cc:408] Connection header received
0202 15:04:24.134578 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:24.134580 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:24.134620 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:24.134713 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:24.135062 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.135856 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.137334 (+ 1478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.137826 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.137860 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:24.137909 (+ 49us) server_negotiation.cc:300] Negotiation successful
0202 15:04:24.137965 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":89,"threads_started":1}
I20260202 15:04:24.137204 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.133596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38971 (local address 127.0.0.1:60016)
0202 15:04:24.134213 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.134227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:24.134308 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:24.134723 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:24.134727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:24.134742 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:24.134942 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.134949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.135984 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.135987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:24.136819 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.136832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.136915 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.136934 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:04:24.136977 (+ 43us) client_negotiation.cc:240] Negotiation successful
0202 15:04:24.137076 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":109,"threads_started":1}
I20260202 15:04:24.138917 10374 tablet_service.cc:1505] Processing CreateTablet for tablet ddda4b1eb2d743879051231cd5a4e74b (DEFAULT_TABLE table=test-table [id=02446352191c4cc1a5456dec9edf0079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:24.139022 10510 tablet_service.cc:1505] Processing CreateTablet for tablet ddda4b1eb2d743879051231cd5a4e74b (DEFAULT_TABLE table=test-table [id=02446352191c4cc1a5456dec9edf0079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:24.139201 10374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddda4b1eb2d743879051231cd5a4e74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:24.139276 10510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddda4b1eb2d743879051231cd5a4e74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:24.139765 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.134248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.2.164.67:45485)
0202 15:04:24.134973 (+ 725us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:24.134979 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:24.134993 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:24.135039 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:24.135042 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:24.135103 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:24.135185 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:24.135574 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.137115 (+ 1541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.138525 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.139257 (+ 732us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.139295 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:24.139350 (+ 55us) server_negotiation.cc:300] Negotiation successful
0202 15:04:24.139412 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":84,"threads_started":1}
I20260202 15:04:24.140009 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.133926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45485 (local address 127.0.0.1:36572)
0202 15:04:24.134551 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.134566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:24.134635 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:24.135210 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:24.135213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:24.135232 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:24.135444 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.135450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.137239 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.137242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:24.138070 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.138092 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.138210 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.138231 (+ 21us) client_negotiation.cc:769] Sending connection context
0202 15:04:24.138275 (+ 44us) client_negotiation.cc:240] Negotiation successful
0202 15:04:24.138325 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":79,"threads_started":1}
I20260202 15:04:24.140612 10646 tablet_service.cc:1505] Processing CreateTablet for tablet ddda4b1eb2d743879051231cd5a4e74b (DEFAULT_TABLE table=test-table [id=02446352191c4cc1a5456dec9edf0079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:24.140874 10646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddda4b1eb2d743879051231cd5a4e74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:24.141918 10798 tablet_bootstrap.cc:492] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396: Bootstrap starting.
I20260202 15:04:24.142232 10799 tablet_bootstrap.cc:492] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282: Bootstrap starting.
I20260202 15:04:24.142868 10798 tablet_bootstrap.cc:654] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:24.143213 10799 tablet_bootstrap.cc:654] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:24.143760 10800 tablet_bootstrap.cc:492] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Bootstrap starting.
I20260202 15:04:24.144375 10799 tablet_bootstrap.cc:492] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282: No bootstrap required, opened a new log
I20260202 15:04:24.144375 10798 tablet_bootstrap.cc:492] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396: No bootstrap required, opened a new log
I20260202 15:04:24.144454 10799 ts_tablet_manager.cc:1403] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:24.144498 10798 ts_tablet_manager.cc:1403] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:24.144891 10798 raft_consensus.cc:359] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.144894 10799 raft_consensus.cc:359] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.145004 10798 raft_consensus.cc:385] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:24.145056 10799 raft_consensus.cc:385] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:24.145077 10798 raft_consensus.cc:740] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604b79fe893d4460ad4c782af7e87396, State: Initialized, Role: FOLLOWER
I20260202 15:04:24.145092 10799 raft_consensus.cc:740] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca6cb50f08848fcad0733a5a39d4282, State: Initialized, Role: FOLLOWER
I20260202 15:04:24.145234 10799 consensus_queue.cc:260] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.145238 10798 consensus_queue.cc:260] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.145563 10799 ts_tablet_manager.cc:1434] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:04:24.145591 10798 ts_tablet_manager.cc:1434] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260202 15:04:24.146291 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.145851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:52193)
0202 15:04:24.146129 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.146185 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260202 15:04:24.146787 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.146183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:35681)
0202 15:04:24.146652 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.146708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":81,"threads_started":1}
I20260202 15:04:24.147634 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.147303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:37505)
0202 15:04:24.147513 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.147566 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260202 15:04:24.148590 10577 heartbeater.cc:499] Master 127.2.164.124:41495 was elected leader, sending a full tablet report...
I20260202 15:04:24.148792 10804 raft_consensus.cc:493] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:24.148885 10804 raft_consensus.cc:515] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.149333 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.147159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:34469)
0202 15:04:24.149205 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.149256 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1959,"thread_start_us":80,"threads_started":1}
I20260202 15:04:24.149477 10441 heartbeater.cc:499] Master 127.2.164.124:41495 was elected leader, sending a full tablet report...
I20260202 15:04:24.150665 10804 leader_election.cc:290] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967), 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485)
I20260202 15:04:24.150771 10800 tablet_bootstrap.cc:654] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:24.152257 10800 tablet_bootstrap.cc:492] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: No bootstrap required, opened a new log
I20260202 15:04:24.152345 10800 ts_tablet_manager.cc:1403] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260202 15:04:24.153721 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.150758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32967 (local address 127.2.164.65:58427)
0202 15:04:24.151003 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.151018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:24.151107 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:24.151438 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:24.151441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:24.151457 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:24.151751 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.151765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.152658 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.152663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:24.153348 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.153356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.153482 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.153501 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:04:24.153555 (+ 54us) client_negotiation.cc:240] Negotiation successful
0202 15:04:24.153616 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260202 15:04:24.154067 10800 raft_consensus.cc:359] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.154165 10800 raft_consensus.cc:385] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:24.154182 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.150951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:58427 (local address 127.2.164.66:32967)
0202 15:04:24.151053 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:24.151058 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:24.151074 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:04:24.151263 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:24.151268 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:24.151324 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:24.151413 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:24.151872 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.152558 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.153475 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.153938 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.153972 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:24.154016 (+ 44us) server_negotiation.cc:300] Negotiation successful
0202 15:04:24.154068 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260202 15:04:24.154203 10800 raft_consensus.cc:740] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea2fcb4eaa94d1086db8301f7f2b3c8, State: Initialized, Role: FOLLOWER
I20260202 15:04:24.154340 10800 consensus_queue.cc:260] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.154650 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.150870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45485 (local address 127.2.164.65:51165)
0202 15:04:24.151549 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.151560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:24.151662 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:24.151917 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:24.151921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:24.151938 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:24.152386 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.152392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.153220 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.153223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:24.153763 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:24.153770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.154389 (+ 619us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.154401 (+ 12us) client_negotiation.cc:769] Sending connection context
0202 15:04:24.154484 (+ 83us) client_negotiation.cc:240] Negotiation successful
0202 15:04:24.154540 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530}
I20260202 15:04:24.154686 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.151120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:51165 (local address 127.2.164.67:45485)
0202 15:04:24.151259 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:24.151264 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:24.151631 (+ 367us) server_negotiation.cc:408] Connection header received
0202 15:04:24.151783 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:24.151785 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:24.151824 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:24.151908 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:24.152504 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.153080 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.153940 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.154267 (+ 327us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.154323 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:24.154520 (+ 197us) server_negotiation.cc:300] Negotiation successful
0202 15:04:24.154589 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260202 15:04:24.154772 10530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddda4b1eb2d743879051231cd5a4e74b" candidate_uuid: "604b79fe893d4460ad4c782af7e87396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca6cb50f08848fcad0733a5a39d4282" is_pre_election: true
I20260202 15:04:24.154683 10800 ts_tablet_manager.cc:1434] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260202 15:04:24.154893 10530 raft_consensus.cc:2468] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604b79fe893d4460ad4c782af7e87396 in term 0.
I20260202 15:04:24.155114 10666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddda4b1eb2d743879051231cd5a4e74b" candidate_uuid: "604b79fe893d4460ad4c782af7e87396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" is_pre_election: true
I20260202 15:04:24.155282 10666 raft_consensus.cc:2468] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604b79fe893d4460ad4c782af7e87396 in term 0.
I20260202 15:04:24.155272 10330 leader_election.cc:304] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604b79fe893d4460ad4c782af7e87396, aca6cb50f08848fcad0733a5a39d4282; no voters:
I20260202 15:04:24.155488 10804 raft_consensus.cc:2804] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:24.155555 10804 raft_consensus.cc:493] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:24.155596 10804 raft_consensus.cc:3060] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:24.156494 10804 raft_consensus.cc:515] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.156809 10804 leader_election.cc:290] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [CANDIDATE]: Term 1 election: Requested vote from peers aca6cb50f08848fcad0733a5a39d4282 (127.2.164.66:32967), 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485)
I20260202 15:04:24.157133 10666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddda4b1eb2d743879051231cd5a4e74b" candidate_uuid: "604b79fe893d4460ad4c782af7e87396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8"
I20260202 15:04:24.157133 10530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddda4b1eb2d743879051231cd5a4e74b" candidate_uuid: "604b79fe893d4460ad4c782af7e87396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca6cb50f08848fcad0733a5a39d4282"
I20260202 15:04:24.157243 10666 raft_consensus.cc:3060] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:24.157251 10530 raft_consensus.cc:3060] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:24.158006 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.155201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:51717)
0202 15:04:24.157871 (+ 2670us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.157925 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2571,"thread_start_us":102,"threads_started":1}
I20260202 15:04:24.158254 10530 raft_consensus.cc:2468] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604b79fe893d4460ad4c782af7e87396 in term 1.
I20260202 15:04:24.158254 10666 raft_consensus.cc:2468] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604b79fe893d4460ad4c782af7e87396 in term 1.
I20260202 15:04:24.158540 10327 leader_election.cc:304] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea2fcb4eaa94d1086db8301f7f2b3c8, 604b79fe893d4460ad4c782af7e87396; no voters:
I20260202 15:04:24.158725 10804 raft_consensus.cc:2804] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:24.158964 10804 raft_consensus.cc:697] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 1 LEADER]: Becoming Leader. State: Replica: 604b79fe893d4460ad4c782af7e87396, State: Running, Role: LEADER
I20260202 15:04:24.159235 10804 consensus_queue.cc:237] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } }
I20260202 15:04:24.159567 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.159374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:50679)
0202 15:04:24.159476 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.159518 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260202 15:04:24.160506 10713 heartbeater.cc:499] Master 127.2.164.124:41495 was elected leader, sending a full tablet report...
I20260202 15:04:24.161564 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.158503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:44797)
0202 15:04:24.161431 (+ 2928us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.161495 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2869,"thread_start_us":86,"threads_started":1}
I20260202 15:04:24.161826 10248 catalog_manager.cc:5654] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 reported cstate change: term changed from 0 to 1, leader changed from <none> to 604b79fe893d4460ad4c782af7e87396 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "604b79fe893d4460ad4c782af7e87396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604b79fe893d4460ad4c782af7e87396" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 } health_report { overall_health: UNKNOWN } } }
I20260202 15:04:24.179729 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.176587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.2.164.65:38971)
0202 15:04:24.176731 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:24.176735 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:24.176748 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:24.176823 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:24.176826 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:24.176887 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:24.176970 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:24.177398 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.177922 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:24.178577 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:24.178720 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:24.178770 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:24.179273 (+ 503us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:24.179370 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:24.179567 (+ 197us) server_negotiation.cc:300] Negotiation successful
0202 15:04:24.179620 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260202 15:04:24.182716 10530 raft_consensus.cc:1275] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Refusing update from remote peer 604b79fe893d4460ad4c782af7e87396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:24.182931 10666 raft_consensus.cc:1275] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Refusing update from remote peer 604b79fe893d4460ad4c782af7e87396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:24.183293 10812 consensus_queue.cc:1048] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca6cb50f08848fcad0733a5a39d4282" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:24.183533 10804 consensus_queue.cc:1048] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea2fcb4eaa94d1086db8301f7f2b3c8" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:24.185107 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.184756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:46807)
0202 15:04:24.184943 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.185027 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260202 15:04:24.186611 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.185364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:46989)
0202 15:04:24.186461 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.186520 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1010}
I20260202 15:04:24.189927 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.189373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.67:43585)
0202 15:04:24.189801 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.189854 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
I20260202 15:04:24.190672 10816 mvcc.cc:204] Tried to move back new op lower bound from 7250102944487170048 to 7250102944398946304. Current Snapshot: MvccSnapshot[applied={T|T < 7250102944487170048}]
I20260202 15:04:24.190846 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.190476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:34397)
0202 15:04:24.190691 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.190746 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260202 15:04:24.191289 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.191044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.66:46789)
0202 15:04:24.191167 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.191219 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260202 15:04:24.192226 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.192002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:45883 (local address 127.2.164.65:44625)
0202 15:04:24.192115 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.192163 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260202 15:04:24.197266 10247 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:24.197391 10247 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:24.200989 10247 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7aea978ffa0f48749984cdc07f255126: Sending DeleteTablet for 3 replicas of tablet ddda4b1eb2d743879051231cd5a4e74b
I20260202 15:04:24.201633 10510 tablet_service.cc:1552] Processing DeleteTablet for tablet ddda4b1eb2d743879051231cd5a4e74b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:24 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:57248
I20260202 15:04:24.201722 10374 tablet_service.cc:1552] Processing DeleteTablet for tablet ddda4b1eb2d743879051231cd5a4e74b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:24 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:60016
I20260202 15:04:24.202261 10646 tablet_service.cc:1552] Processing DeleteTablet for tablet ddda4b1eb2d743879051231cd5a4e74b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:24 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:36572
I20260202 15:04:24.202497 10827 tablet_replica.cc:333] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396: stopping tablet replica
I20260202 15:04:24.202561 10828 tablet_replica.cc:333] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: stopping tablet replica
I20260202 15:04:24.202672 10828 raft_consensus.cc:2243] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:24.202688 10827 raft_consensus.cc:2243] T ddda4b1eb2d743879051231cd5a4e74b P 604b79fe893d4460ad4c782af7e87396 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:24.202809 10828 raft_consensus.cc:2272] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:24.202908 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 10311
I20260202 15:04:24.204883 10828 ts_tablet_manager.cc:1916] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:24.206789 10828 ts_tablet_manager.cc:1929] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:24.206887 10828 log.cc:1199] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/ddda4b1eb2d743879051231cd5a4e74b
I20260202 15:04:24.207181 10828 ts_tablet_manager.cc:1950] T ddda4b1eb2d743879051231cd5a4e74b P 2ea2fcb4eaa94d1086db8301f7f2b3c8: Deleting consensus metadata
I20260202 15:04:24.207792 10232 catalog_manager.cc:4985] TS 2ea2fcb4eaa94d1086db8301f7f2b3c8 (127.2.164.67:45485): tablet ddda4b1eb2d743879051231cd5a4e74b (table test-table [id=02446352191c4cc1a5456dec9edf0079]) successfully deleted
I20260202 15:04:24.209233 10826 tablet_replica.cc:333] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282: stopping tablet replica
I20260202 15:04:24.209385 10826 raft_consensus.cc:2243] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:24.209538 10826 raft_consensus.cc:2272] T ddda4b1eb2d743879051231cd5a4e74b P aca6cb50f08848fcad0733a5a39d4282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:24.210109 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 10447
I20260202 15:04:24.211118 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.210272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38971 (local address 127.0.0.1:60038)
0202 15:04:24.210999 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.211045 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635}
W20260202 15:04:24.211274 10233 catalog_manager.cc:4712] TS 604b79fe893d4460ad4c782af7e87396 (127.2.164.65:38971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddda4b1eb2d743879051231cd5a4e74b: Network error: Client connection negotiation failed: client connection to 127.2.164.65:38971: connect: Connection refused (error 111)
I20260202 15:04:24.217408 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 10583
I20260202 15:04:24.218248 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:24.217501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32967 (local address 127.0.0.1:57264)
0202 15:04:24.218090 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:24.218147 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.66:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480}
I20260202 15:04:24.224157 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 9924
I20260202 15:04:24.230074 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 10045
2026-02-02T15:04:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7407 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:24 dist-test-slave-6zzp krb5kdc[10831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:24 dist-test-slave-6zzp krb5kdc[10831](info): set up 2 sockets
Feb 02 15:04:24 dist-test-slave-6zzp krb5kdc[10831](info): commencing operation
krb5kdc: starting...
W20260202 15:04:26.249802 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:26 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:26.392343 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41155
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:26.504480 10847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:26.504772 10847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:26.504863 10847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:26.508246 10847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:26.508356 10847 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260202 15:04:26.508390 10847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:26.508442 10847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:26.508488 10847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:26.508546 10847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:26.513020 10847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:26.514214 10847 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:26.515108 10847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:26.521065 10852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:26.521065 10853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:26.521088 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:26.521247 10847 server_base.cc:1060] running on GCE node
I20260202 15:04:26.522002 10847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:26.522926 10847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:26.524102 10847 hybrid_clock.cc:648] HybridClock initialized: now 1770044666524091 us; error 28 us; skew 500 ppm
Feb 02 15:04:26 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044666, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:26.526803 10847 init.cc:377] Logged in from keytab as oryx/127.2.164.126@KRBTEST.COM (short username oryx)
I20260202 15:04:26.527918 10847 webserver.cc:492] Webserver started at http://127.2.164.126:36705/ using document root <none> and password file <none>
I20260202 15:04:26.528440 10847 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:26.528488 10847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:26.528645 10847 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:26.530355 10847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "165f322e88af4b839a53e3c7b0bfb936"
format_stamp: "Formatted at 2026-02-02 15:04:26 on dist-test-slave-6zzp"
server_key: "f402abaf7dde53a1ff0386f6a1a4aa15"
server_key_iv: "3929b6cdacc40a3f2dc7501f35d3905a"
server_key_version: "encryptionkey@0"
I20260202 15:04:26.530790 10847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "165f322e88af4b839a53e3c7b0bfb936"
format_stamp: "Formatted at 2026-02-02 15:04:26 on dist-test-slave-6zzp"
server_key: "f402abaf7dde53a1ff0386f6a1a4aa15"
server_key_iv: "3929b6cdacc40a3f2dc7501f35d3905a"
server_key_version: "encryptionkey@0"
I20260202 15:04:26.534409 10847 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:26.536772 10862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:26.537974 10847 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:26.538121 10847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "165f322e88af4b839a53e3c7b0bfb936"
format_stamp: "Formatted at 2026-02-02 15:04:26 on dist-test-slave-6zzp"
server_key: "f402abaf7dde53a1ff0386f6a1a4aa15"
server_key_iv: "3929b6cdacc40a3f2dc7501f35d3905a"
server_key_version: "encryptionkey@0"
I20260202 15:04:26.538236 10847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:26.570330 10847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:26.573323 10847 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:26.573539 10847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:26.582935 10847 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:41155
I20260202 15:04:26.583276 10924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:41155 every 8 connection(s)
I20260202 15:04:26.584074 10847 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:26.587834 10925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:26.588958 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 10847
I20260202 15:04:26.589171 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:26.589459 2705 external_mini_cluster.cc:1468] Setting key de28818557f4798bd529acdc8b8e803f
I20260202 15:04:26.594362 10925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Bootstrap starting.
I20260202 15:04:26.596737 10925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Neither blocks nor log segments found. Creating new log.
Feb 02 15:04:26 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:26.597690 10925 log.cc:826] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:26.599918 10925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: No bootstrap required, opened a new log
I20260202 15:04:26.603191 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:26.590791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.2.164.126:41155)
0202 15:04:26.591246 (+ 455us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:26.591258 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:26.591301 (+ 43us) server_negotiation.cc:408] Connection header received
0202 15:04:26.591965 (+ 664us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:26.591990 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:26.592349 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:26.592711 (+ 362us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:26.593614 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.594732 (+ 1118us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:26.595477 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.595756 (+ 279us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:26.598223 (+ 2467us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:26.598244 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:26.598260 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:26.598300 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:26.600453 (+ 2153us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:26.600955 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:26.600960 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:26.600964 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:26.601101 (+ 137us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:26.601377 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:26.601380 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:26.601381 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:26.601731 (+ 350us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:26.601929 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:26.602246 (+ 317us) server_negotiation.cc:300] Negotiation successful
0202 15:04:26.602513 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":143,"threads_started":1}
I20260202 15:04:26.603583 10925 raft_consensus.cc:359] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165f322e88af4b839a53e3c7b0bfb936" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41155 } }
I20260202 15:04:26.603770 10925 raft_consensus.cc:385] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:26.603832 10925 raft_consensus.cc:740] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165f322e88af4b839a53e3c7b0bfb936, State: Initialized, Role: FOLLOWER
I20260202 15:04:26.604290 10925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165f322e88af4b839a53e3c7b0bfb936" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41155 } }
I20260202 15:04:26.604408 10925 raft_consensus.cc:399] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:26.604491 10925 raft_consensus.cc:493] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:26.604583 10925 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:26.605535 10925 raft_consensus.cc:515] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165f322e88af4b839a53e3c7b0bfb936" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41155 } }
I20260202 15:04:26.605880 10925 leader_election.cc:304] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165f322e88af4b839a53e3c7b0bfb936; no voters:
I20260202 15:04:26.606175 10925 leader_election.cc:290] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:26.606338 10930 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:26.606560 10930 raft_consensus.cc:697] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [term 1 LEADER]: Becoming Leader. State: Replica: 165f322e88af4b839a53e3c7b0bfb936, State: Running, Role: LEADER
I20260202 15:04:26.606859 10930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165f322e88af4b839a53e3c7b0bfb936" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41155 } }
I20260202 15:04:26.607368 10925 sys_catalog.cc:565] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:26.608466 10932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165f322e88af4b839a53e3c7b0bfb936. Latest consensus state: current_term: 1 leader_uuid: "165f322e88af4b839a53e3c7b0bfb936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165f322e88af4b839a53e3c7b0bfb936" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41155 } } }
I20260202 15:04:26.608561 10932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:26.608440 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165f322e88af4b839a53e3c7b0bfb936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165f322e88af4b839a53e3c7b0bfb936" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41155 } } }
I20260202 15:04:26.608888 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:26.609028 10939 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:26.612048 10939 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:26.617619 10939 catalog_manager.cc:1357] Generated new cluster ID: 4da840d9c65f4bc895e64bb28318239e
I20260202 15:04:26.617704 10939 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:26.629369 10939 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:26.630409 10939 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:26.637408 10939 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Generated new TSK 0
I20260202 15:04:26.638357 10939 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:26.696156 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:41155
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:26.808915 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:26.809160 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:26.809262 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:26.812701 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:26.812770 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:26.812860 10953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:04:26.812942 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:26.817222 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:26.818348 10953 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:26.819478 10953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:26.826548 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:26.826522 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:26.826522 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:26.826642 10953 server_base.cc:1060] running on GCE node
I20260202 15:04:26.827009 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:26.827556 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:26.828728 10953 hybrid_clock.cc:648] HybridClock initialized: now 1770044666828719 us; error 33 us; skew 500 ppm
Feb 02 15:04:26 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044666, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:26.831364 10953 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260202 15:04:26.832568 10953 webserver.cc:492] Webserver started at http://127.2.164.65:34927/ using document root <none> and password file <none>
I20260202 15:04:26.833237 10953 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:26.833316 10953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:26.833544 10953 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:26.835330 10953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "aba950577ef745938c0372f0c49667fe"
format_stamp: "Formatted at 2026-02-02 15:04:26 on dist-test-slave-6zzp"
server_key: "aa1be565f0d5fc65639efeb7a107ec01"
server_key_iv: "8f59c43c2dbe1727c33b3c628b8dc780"
server_key_version: "encryptionkey@0"
I20260202 15:04:26.835866 10953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "aba950577ef745938c0372f0c49667fe"
format_stamp: "Formatted at 2026-02-02 15:04:26 on dist-test-slave-6zzp"
server_key: "aa1be565f0d5fc65639efeb7a107ec01"
server_key_iv: "8f59c43c2dbe1727c33b3c628b8dc780"
server_key_version: "encryptionkey@0"
I20260202 15:04:26.839661 10953 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260202 15:04:26.842118 10968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:26.843286 10953 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:26.843389 10953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "aba950577ef745938c0372f0c49667fe"
format_stamp: "Formatted at 2026-02-02 15:04:26 on dist-test-slave-6zzp"
server_key: "aa1be565f0d5fc65639efeb7a107ec01"
server_key_iv: "8f59c43c2dbe1727c33b3c628b8dc780"
server_key_version: "encryptionkey@0"
I20260202 15:04:26.843479 10953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:26.869339 10953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:26.876866 10953 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:26.877135 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:26.878719 10953 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:26.878813 10953 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:26.878921 10953 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:26.878965 10953 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:04:26 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044666, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:26.890735 10953 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:38313
I20260202 15:04:26.891216 11083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:38313 every 8 connection(s)
I20260202 15:04:26.892005 10953 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:26.892889 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:26.879559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:42201 (local address 127.2.164.126:41155)
0202 15:04:26.879692 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:26.879696 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:26.880547 (+ 851us) server_negotiation.cc:408] Connection header received
0202 15:04:26.881529 (+ 982us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:26.881533 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:26.881592 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:26.881699 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:26.883240 (+ 1541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.883981 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:26.884994 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.885248 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:26.888111 (+ 2863us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:26.888143 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:26.888150 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:26.888197 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:26.890027 (+ 1830us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:26.890658 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:26.890662 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:26.890664 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:26.890719 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:26.891083 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:26.891087 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:26.891089 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:26.891306 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:26.891406 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:26.892235 (+ 829us) server_negotiation.cc:300] Negotiation successful
0202 15:04:26.892730 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260202 15:04:26.893167 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:26.879873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.2.164.65:42201)
0202 15:04:26.880390 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:26.880427 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:26.881323 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:26.881869 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:26.881878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:26.882468 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:26.883081 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:26.883094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.884128 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:26.884135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:26.884852 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:26.884865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.885171 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:26.885899 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:26.885936 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:26.887921 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:26.890162 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:26.890167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:26.890194 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:26.890526 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:26.890835 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:26.890841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:26.890846 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:26.890965 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:26.891466 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:26.891472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:26.891770 (+ 298us) client_negotiation.cc:769] Sending connection context
0202 15:04:26.892008 (+ 238us) client_negotiation.cc:240] Negotiation successful
0202 15:04:26.892323 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":174,"threads_started":1}
I20260202 15:04:26.893635 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 10953
I20260202 15:04:26.893783 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:26.894078 2705 external_mini_cluster.cc:1468] Setting key 8031cf4fdaffd64f49b4d49d8b2dc62b
I20260202 15:04:26.894534 11084 heartbeater.cc:344] Connected to a master server at 127.2.164.126:41155
I20260202 15:04:26.894690 11084 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:26.895201 11084 heartbeater.cc:507] Master 127.2.164.126:41155 requested a full tablet report, sending...
I20260202 15:04:26.896801 10878 ts_manager.cc:194] Registered new tserver with Master: aba950577ef745938c0372f0c49667fe (127.2.164.65:38313)
I20260202 15:04:26.898504 10878 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:42201
I20260202 15:04:26.904175 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:26.896885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:55135 (local address 127.2.164.126:41155)
0202 15:04:26.897031 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:26.897035 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:26.897310 (+ 275us) server_negotiation.cc:408] Connection header received
0202 15:04:26.897453 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:26.897457 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:26.897504 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:26.897590 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:26.898300 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.898969 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:26.899915 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.900108 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:26.901175 (+ 1067us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:26.901191 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:26.901193 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:26.901215 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:26.902611 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:26.903110 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:26.903113 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:26.903114 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:26.903156 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:26.903429 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:26.903431 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:26.903432 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:26.903569 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:26.903644 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:26.903939 (+ 295us) server_negotiation.cc:300] Negotiation successful
0202 15:04:26.904051 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260202 15:04:26.904277 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:26.896818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.2.164.65:55135)
0202 15:04:26.897206 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:26.897225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:26.897330 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:26.897624 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:26.897630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:26.897880 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:26.898168 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:26.898176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.899132 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:26.899139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:26.899771 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:26.899784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:26.899931 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:26.900536 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:26.900558 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:26.901022 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:26.902735 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:26.902739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:26.902742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:26.902998 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:26.903257 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:26.903262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:26.903265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:26.903335 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:26.903667 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:26.903672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:26.903779 (+ 107us) client_negotiation.cc:769] Sending connection context
0202 15:04:26.903895 (+ 116us) client_negotiation.cc:240] Negotiation successful
0202 15:04:26.904060 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:26.946168 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:41155
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:27.057545 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:27.057777 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:27.057839 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:27.061112 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:27.061203 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:27.061262 11095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:04:27.061311 11095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:27.065518 11095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:27.066560 11095 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:27.067358 11095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:27.073738 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:27.073738 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:27.073747 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:27.074177 11095 server_base.cc:1060] running on GCE node
I20260202 15:04:27.074487 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:27.075013 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:27.076161 11095 hybrid_clock.cc:648] HybridClock initialized: now 1770044667076138 us; error 37 us; skew 500 ppm
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:27.078706 11095 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260202 15:04:27.079715 11095 webserver.cc:492] Webserver started at http://127.2.164.66:33977/ using document root <none> and password file <none>
I20260202 15:04:27.080220 11095 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:27.080274 11095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:27.080428 11095 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:27.082163 11095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "ccb7b766b7c04da3bbaff1ad468894ff"
format_stamp: "Formatted at 2026-02-02 15:04:27 on dist-test-slave-6zzp"
server_key: "7686549a5deca00486a4f23d175ec2df"
server_key_iv: "7f84823feca75fcdfb272d6d6f0cc7a1"
server_key_version: "encryptionkey@0"
I20260202 15:04:27.082600 11095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "ccb7b766b7c04da3bbaff1ad468894ff"
format_stamp: "Formatted at 2026-02-02 15:04:27 on dist-test-slave-6zzp"
server_key: "7686549a5deca00486a4f23d175ec2df"
server_key_iv: "7f84823feca75fcdfb272d6d6f0cc7a1"
server_key_version: "encryptionkey@0"
I20260202 15:04:27.085889 11095 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:27.087960 11110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:27.088973 11095 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:27.089159 11095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "ccb7b766b7c04da3bbaff1ad468894ff"
format_stamp: "Formatted at 2026-02-02 15:04:27 on dist-test-slave-6zzp"
server_key: "7686549a5deca00486a4f23d175ec2df"
server_key_iv: "7f84823feca75fcdfb272d6d6f0cc7a1"
server_key_version: "encryptionkey@0"
I20260202 15:04:27.089274 11095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:27.105535 11095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:27.108167 11095 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:27.108421 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:27.110087 11095 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:27.110162 11095 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:27.110248 11095 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:27.110298 11095 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:27.121224 11095 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:36097
I20260202 15:04:27.121276 11225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:36097 every 8 connection(s)
I20260202 15:04:27.122378 11095 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:27.123464 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.111013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.2.164.66:49281)
0202 15:04:27.111460 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.111502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.112343 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.113021 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.113032 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.113634 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:27.114368 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.114401 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.115500 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.115503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.116161 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.116173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.116404 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.117046 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:27.117089 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:27.118830 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:27.120949 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.120956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.120967 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.121319 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.121542 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.121545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.121547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.121639 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.121982 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:27.121988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:27.122262 (+ 274us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.122473 (+ 211us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.122738 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
I20260202 15:04:27.124073 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.110725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:49281 (local address 127.2.164.126:41155)
0202 15:04:27.110881 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.110887 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.111610 (+ 723us) server_negotiation.cc:408] Connection header received
0202 15:04:27.112637 (+ 1027us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.112643 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.112707 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.112814 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.114541 (+ 1727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.115366 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.116293 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.116530 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.119031 (+ 2501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.119055 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.119059 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.119095 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.120804 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.121416 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.121419 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.121420 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.121464 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.121737 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.121739 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.121740 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.121875 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.121948 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.123833 (+ 1885us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.123960 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260202 15:04:27.124918 11226 heartbeater.cc:344] Connected to a master server at 127.2.164.126:41155
I20260202 15:04:27.125123 11226 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:27.125658 11226 heartbeater.cc:507] Master 127.2.164.126:41155 requested a full tablet report, sending...
I20260202 15:04:27.126677 10879 ts_manager.cc:194] Registered new tserver with Master: ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097)
I20260202 15:04:27.127348 10879 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:49281
I20260202 15:04:27.132413 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 11095
I20260202 15:04:27.132561 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:27.132891 2705 external_mini_cluster.cc:1468] Setting key 5cac7eb077c68a2eac8ed8173d74e8f5
I20260202 15:04:27.133759 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.127056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:33373 (local address 127.2.164.126:41155)
0202 15:04:27.127194 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.127204 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.127781 (+ 577us) server_negotiation.cc:408] Connection header received
0202 15:04:27.127943 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.127946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.128003 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.128067 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.128733 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.129285 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.129890 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.130026 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.130830 (+ 804us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.130844 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.130846 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.130863 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.132012 (+ 1149us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.132479 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.132481 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.132483 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.132558 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.132830 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.132832 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.132834 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.133108 (+ 274us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.133190 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.133504 (+ 314us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.133614 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:04:27.133777 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.126984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.2.164.66:33373)
0202 15:04:27.127692 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.127705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.127845 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.128122 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.128125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.128318 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:27.128598 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.128607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.129392 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.129395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.129794 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.129802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.129894 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.130434 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:27.130447 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:27.130731 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:27.132109 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.132112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.132113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.132381 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.132671 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.132674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.132676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.132738 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.133215 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:27.133218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:27.133310 (+ 92us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.133448 (+ 138us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.133590 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab.
I20260202 15:04:27.184387 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:41155
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:27.297361 11237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:27.297598 11237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:27.297696 11237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:27.301048 11237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:27.301117 11237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:27.301221 11237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260202 15:04:27.301291 11237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:27.305831 11237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/krb5kdc/oryx.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:27.307113 11237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:27.308045 11237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:27.315060 11243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:27.315060 11242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:27.315250 11245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:27.315299 11237 server_base.cc:1060] running on GCE node
I20260202 15:04:27.315713 11237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:27.316270 11237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:27.317416 11237 hybrid_clock.cc:648] HybridClock initialized: now 1770044667317392 us; error 39 us; skew 500 ppm
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:27.320025 11237 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260202 15:04:27.321200 11237 webserver.cc:492] Webserver started at http://127.2.164.67:37771/ using document root <none> and password file <none>
I20260202 15:04:27.321727 11237 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:27.321772 11237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:27.321926 11237 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:27.323583 11237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "6845808a908d4457bedf735367f92b97"
format_stamp: "Formatted at 2026-02-02 15:04:27 on dist-test-slave-6zzp"
server_key: "a4b5722ea5678bea1c5d5ede277d6d1e"
server_key_iv: "89fd6ddea8033c3c18f843570807197f"
server_key_version: "encryptionkey@0"
I20260202 15:04:27.324012 11237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "6845808a908d4457bedf735367f92b97"
format_stamp: "Formatted at 2026-02-02 15:04:27 on dist-test-slave-6zzp"
server_key: "a4b5722ea5678bea1c5d5ede277d6d1e"
server_key_iv: "89fd6ddea8033c3c18f843570807197f"
server_key_version: "encryptionkey@0"
I20260202 15:04:27.327590 11237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260202 15:04:27.329804 11252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:27.331072 11237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260202 15:04:27.331214 11237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "6845808a908d4457bedf735367f92b97"
format_stamp: "Formatted at 2026-02-02 15:04:27 on dist-test-slave-6zzp"
server_key: "a4b5722ea5678bea1c5d5ede277d6d1e"
server_key_iv: "89fd6ddea8033c3c18f843570807197f"
server_key_version: "encryptionkey@0"
I20260202 15:04:27.331324 11237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:27.351351 11237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:27.354509 11237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:27.354727 11237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:27.356477 11237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:27.356559 11237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:27.356626 11237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:27.356664 11237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:27.368335 11237 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:42011
I20260202 15:04:27.368350 11367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:42011 every 8 connection(s)
I20260202 15:04:27.369663 11237 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:27.370630 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 11237
I20260202 15:04:27.370774 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:27.370981 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.357556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.2.164.67:47815)
0202 15:04:27.358084 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.358136 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.359140 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.359744 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.359758 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.360408 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:27.361248 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.361269 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.362450 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.362454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.363126 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.363135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.363378 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.364039 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:27.364078 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:27.365793 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:27.368058 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.368063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.368075 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.368404 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.368682 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.368686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.368688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.368787 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.369328 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:27.369338 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:27.369619 (+ 281us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.369838 (+ 219us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.370127 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260202 15:04:27.371068 2705 external_mini_cluster.cc:1468] Setting key 8e9f58048f4da1c0367774f40d574734
I20260202 15:04:27.371613 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.357250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:47815 (local address 127.2.164.126:41155)
0202 15:04:27.357402 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.357407 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.358267 (+ 860us) server_negotiation.cc:408] Connection header received
0202 15:04:27.359373 (+ 1106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.359377 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.359426 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.359550 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.361458 (+ 1908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.362317 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.363271 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.363469 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.365974 (+ 2505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.366001 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.366009 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.366047 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.367831 (+ 1784us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.368536 (+ 705us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.368540 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.368541 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.368587 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.368946 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.368949 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.368951 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.369209 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.369299 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.371357 (+ 2058us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.371486 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:04:27.373062 11368 heartbeater.cc:344] Connected to a master server at 127.2.164.126:41155
I20260202 15:04:27.373323 11368 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:27.373823 11368 heartbeater.cc:507] Master 127.2.164.126:41155 requested a full tablet report, sending...
I20260202 15:04:27.374770 10878 ts_manager.cc:194] Registered new tserver with Master: 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:27.375332 10878 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:47815
I20260202 15:04:27.381376 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.374451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.2.164.67:40051)
0202 15:04:27.375722 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.375735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.375844 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.376082 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.376087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.376282 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:27.376547 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.376555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.377298 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.377302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.377731 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.377739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.377844 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.378353 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:27.378366 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:27.378637 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:27.380248 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.380251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.380252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.380404 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.380594 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.380597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.380598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.380636 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.380897 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:27.380899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:27.380983 (+ 84us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.381143 (+ 160us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.381242 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":105,"threads_started":1}
I20260202 15:04:27.381398 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.374525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40051 (local address 127.2.164.126:41155)
0202 15:04:27.374641 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.374647 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.375808 (+ 1161us) server_negotiation.cc:408] Connection header received
0202 15:04:27.375932 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.375935 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.375975 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.376042 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.376661 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.377186 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.377855 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.377978 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.378749 (+ 771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.378764 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.378766 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.378787 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.380140 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.380482 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.380484 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.380486 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.380523 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.380705 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.380707 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.380708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.380826 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.380897 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.381194 (+ 297us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.381288 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260202 15:04:27.386137 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260202 15:04:27.408476 10928 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:27.402850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.2.164.126:41155)
0202 15:04:27.403007 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.403011 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.403023 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:27.403063 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.403065 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.403106 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.403214 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.404017 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.404558 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.405264 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.405395 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.408298 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51502: BlockingRecv error: recv got EOF from 127.0.0.1:51502 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:27.416803 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.409996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.2.164.126:41155)
0202 15:04:27.410121 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.410124 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.410168 (+ 44us) server_negotiation.cc:408] Connection header received
0202 15:04:27.410282 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.410285 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.410325 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.410396 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.411055 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.411517 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.412127 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.412253 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.414311 (+ 2058us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.414327 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.414329 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.414352 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.415691 (+ 1339us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.416004 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.416006 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.416008 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.416060 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.416224 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.416226 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.416227 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.416357 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.416453 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.416597 (+ 144us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.416682 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260202 15:04:27.419255 10878 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:04:27.421653 10878 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:27.433789 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.429434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38313 (local address 127.0.0.1:59862)
0202 15:04:27.429791 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.429829 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.430032 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.430711 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.430719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.430741 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.431068 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.431080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.432478 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.432484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.433301 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.433310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.433421 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.433474 (+ 53us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.433560 (+ 86us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.433638 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260202 15:04:27.434157 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.429535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42011 (local address 127.0.0.1:55318)
0202 15:04:27.430559 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.430573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.430666 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.431310 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.431313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.431333 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.431565 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.431571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.432880 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.432883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.433766 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.433775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.433879 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.433898 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.433950 (+ 52us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.434025 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":96,"threads_started":1}
I20260202 15:04:27.434896 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.429538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.2.164.65:38313)
0202 15:04:27.430229 (+ 691us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.430236 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.430262 (+ 26us) server_negotiation.cc:408] Connection header received
0202 15:04:27.430352 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.430357 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.430541 (+ 184us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.431207 (+ 666us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.431341 (+ 134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.432370 (+ 1029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.433943 (+ 1573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.434473 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.434588 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.434647 (+ 59us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.434726 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":77,"threads_started":1}
I20260202 15:04:27.435106 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.430183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.2.164.67:42011)
0202 15:04:27.430548 (+ 365us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.430555 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.430756 (+ 201us) server_negotiation.cc:408] Connection header received
0202 15:04:27.430839 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.430844 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.430991 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.431127 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.431690 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.432758 (+ 1068us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.434345 (+ 1587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.434711 (+ 366us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.434817 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.434883 (+ 66us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.434963 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260202 15:04:27.435300 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.430267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36097 (local address 127.0.0.1:41140)
0202 15:04:27.431137 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.431149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.431231 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.431880 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.431884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.431903 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.432131 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.432137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.433424 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.433429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.434719 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.434728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.434845 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.434861 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.434912 (+ 51us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.435143 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":135,"threads_started":1}
I20260202 15:04:27.436401 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.430696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.2.164.66:36097)
0202 15:04:27.431483 (+ 787us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.431489 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.431506 (+ 17us) server_negotiation.cc:408] Connection header received
0202 15:04:27.431566 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.431570 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.431737 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.431880 (+ 143us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.432251 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.433251 (+ 1000us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.434876 (+ 1625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.436046 (+ 1170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.436161 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.436227 (+ 66us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.436294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":119,"threads_started":1}
I20260202 15:04:27.437950 11018 tablet_service.cc:1505] Processing CreateTablet for tablet d8e936d79a8d44189a25e31da89b204a (DEFAULT_TABLE table=test-table [id=c7f3421db9ea4c888f3199e512720f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.437950 11302 tablet_service.cc:1505] Processing CreateTablet for tablet d8e936d79a8d44189a25e31da89b204a (DEFAULT_TABLE table=test-table [id=c7f3421db9ea4c888f3199e512720f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.438705 11160 tablet_service.cc:1505] Processing CreateTablet for tablet d8e936d79a8d44189a25e31da89b204a (DEFAULT_TABLE table=test-table [id=c7f3421db9ea4c888f3199e512720f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.438963 11302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e936d79a8d44189a25e31da89b204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.438969 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e936d79a8d44189a25e31da89b204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.439451 11160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8e936d79a8d44189a25e31da89b204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.444726 11396 tablet_bootstrap.cc:492] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Bootstrap starting.
I20260202 15:04:27.445988 11397 tablet_bootstrap.cc:492] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Bootstrap starting.
I20260202 15:04:27.447325 11396 tablet_bootstrap.cc:654] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.448050 11397 tablet_bootstrap.cc:654] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.448436 11398 tablet_bootstrap.cc:492] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Bootstrap starting.
I20260202 15:04:27.448678 11396 log.cc:826] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:27.449759 11397 log.cc:826] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:27.450344 11398 tablet_bootstrap.cc:654] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.450709 11396 tablet_bootstrap.cc:492] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: No bootstrap required, opened a new log
I20260202 15:04:27.450942 11396 ts_tablet_manager.cc:1403] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:04:27.451427 11397 tablet_bootstrap.cc:492] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: No bootstrap required, opened a new log
I20260202 15:04:27.451603 11397 ts_tablet_manager.cc:1403] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260202 15:04:27.451731 11398 log.cc:826] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:27.453969 11398 tablet_bootstrap.cc:492] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: No bootstrap required, opened a new log
I20260202 15:04:27.454159 11398 ts_tablet_manager.cc:1403] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:04:27.454324 11397 raft_consensus.cc:359] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.454530 11397 raft_consensus.cc:385] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.454582 11397 raft_consensus.cc:740] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba950577ef745938c0372f0c49667fe, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.454532 11396 raft_consensus.cc:359] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.454736 11396 raft_consensus.cc:385] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.454797 11396 raft_consensus.cc:740] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.455236 11396 consensus_queue.cc:260] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.455276 11397 consensus_queue.cc:260] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.455909 11226 heartbeater.cc:499] Master 127.2.164.126:41155 was elected leader, sending a full tablet report...
I20260202 15:04:27.456123 11397 ts_tablet_manager.cc:1434] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260202 15:04:27.456125 11396 ts_tablet_manager.cc:1434] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260202 15:04:27.456437 11084 heartbeater.cc:499] Master 127.2.164.126:41155 was elected leader, sending a full tablet report...
I20260202 15:04:27.457310 11398 raft_consensus.cc:359] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.457525 11398 raft_consensus.cc:385] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.457587 11398 raft_consensus.cc:740] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6845808a908d4457bedf735367f92b97, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.458182 11398 consensus_queue.cc:260] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.459053 11398 ts_tablet_manager.cc:1434] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260202 15:04:27.459230 11368 heartbeater.cc:499] Master 127.2.164.126:41155 was elected leader, sending a full tablet report...
I20260202 15:04:27.480261 11402 raft_consensus.cc:493] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:27.480474 11402 raft_consensus.cc:515] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.481725 11402 leader_election.cc:290] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:27.484834 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.481994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42011 (local address 127.2.164.66:55243)
0202 15:04:27.482111 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.482127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.482260 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.482515 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.482520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.482544 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.482767 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.482774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.483764 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.483768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.484531 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.484539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.484624 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.484642 (+ 18us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.484678 (+ 36us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.484724 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260202 15:04:27.485463 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.482105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:55243 (local address 127.2.164.67:42011)
0202 15:04:27.482274 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.482279 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.482304 (+ 25us) server_negotiation.cc:408] Connection header received
0202 15:04:27.482352 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.482354 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.482395 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.482468 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.482917 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.483650 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.484659 (+ 1009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.485165 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.485217 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.485268 (+ 51us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.485350 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260202 15:04:27.485522 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.481994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38313 (local address 127.2.164.66:32825)
0202 15:04:27.482684 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.482696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.482808 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.483102 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.483105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.483123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.483425 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.483432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.484307 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.484310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.485135 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.485147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.485268 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.485284 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.485334 (+ 50us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.485383 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":340,"thread_start_us":167,"threads_started":1}
I20260202 15:04:27.485947 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.482201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:32825 (local address 127.2.164.65:38313)
0202 15:04:27.482371 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.482377 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.482782 (+ 405us) server_negotiation.cc:408] Connection header received
0202 15:04:27.482945 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.482949 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.482992 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.483052 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.483532 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.484192 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.485280 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.485723 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.485757 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.485806 (+ 49us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.485861 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260202 15:04:27.486147 11320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8e936d79a8d44189a25e31da89b204a" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6845808a908d4457bedf735367f92b97" is_pre_election: true
I20260202 15:04:27.486505 11320 raft_consensus.cc:2468] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 0.
I20260202 15:04:27.487052 11114 leader_election.cc:304] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:27.487061 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8e936d79a8d44189a25e31da89b204a" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe" is_pre_election: true
I20260202 15:04:27.487315 11038 raft_consensus.cc:2468] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 0.
I20260202 15:04:27.487319 11402 raft_consensus.cc:2804] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:27.487425 11402 raft_consensus.cc:493] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:27.487473 11402 raft_consensus.cc:3060] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.488415 11402 raft_consensus.cc:515] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.488727 11402 leader_election.cc:290] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 election: Requested vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:27.489231 11320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8e936d79a8d44189a25e31da89b204a" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6845808a908d4457bedf735367f92b97"
I20260202 15:04:27.489315 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8e936d79a8d44189a25e31da89b204a" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe"
I20260202 15:04:27.489387 11320 raft_consensus.cc:3060] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.489456 11038 raft_consensus.cc:3060] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.490531 11038 raft_consensus.cc:2468] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 1.
I20260202 15:04:27.490648 11320 raft_consensus.cc:2468] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 1.
I20260202 15:04:27.490929 11113 leader_election.cc:304] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba950577ef745938c0372f0c49667fe, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:27.491110 11402 raft_consensus.cc:2804] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:27.491374 11402 raft_consensus.cc:697] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 1 LEADER]: Becoming Leader. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Running, Role: LEADER
I20260202 15:04:27.491894 11402 consensus_queue.cc:237] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } }
I20260202 15:04:27.495050 10878 catalog_manager.cc:5654] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } health_report { overall_health: HEALTHY } } }
I20260202 15:04:27.514693 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.511673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.2.164.66:36097)
0202 15:04:27.511812 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.511815 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.511828 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:27.511901 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.511903 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.511957 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.512058 (+ 101us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:27.512353 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.512838 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.513487 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.513705 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.513770 (+ 65us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:27.514285 (+ 515us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:27.514376 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.514458 (+ 82us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.514526 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260202 15:04:27.519801 11320 raft_consensus.cc:1275] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Refusing update from remote peer ccb7b766b7c04da3bbaff1ad468894ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:27.519802 11038 raft_consensus.cc:1275] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Refusing update from remote peer ccb7b766b7c04da3bbaff1ad468894ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:27.520411 11402 consensus_queue.cc:1048] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:27.520560 11406 consensus_queue.cc:1048] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:27.527127 11409 mvcc.cc:204] Tried to move back new op lower bound from 7250102958154444800 to 7250102958049210368. Current Snapshot: MvccSnapshot[applied={T|T < 7250102958154444800}]
I20260202 15:04:27.528733 11410 mvcc.cc:204] Tried to move back new op lower bound from 7250102958154444800 to 7250102958049210368. Current Snapshot: MvccSnapshot[applied={T|T < 7250102958154444800}]
I20260202 15:04:27.537757 10878 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:27.537969 10878 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:27.540518 10878 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Sending DeleteTablet for 3 replicas of tablet d8e936d79a8d44189a25e31da89b204a
I20260202 15:04:27.541508 11160 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e936d79a8d44189a25e31da89b204a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:27 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:41140
I20260202 15:04:27.541512 11018 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e936d79a8d44189a25e31da89b204a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:27 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:59862
I20260202 15:04:27.541512 11302 tablet_service.cc:1552] Processing DeleteTablet for tablet d8e936d79a8d44189a25e31da89b204a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:27 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260202 15:04:27.542457 11420 tablet_replica.cc:333] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: stopping tablet replica
I20260202 15:04:27.542789 11420 raft_consensus.cc:2243] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:27.542915 10878 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260202 15:04:27.543110 11420 raft_consensus.cc:2272] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260202 15:04:27.543416 10878 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:27.543434 11421 tablet_replica.cc:333] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: stopping tablet replica
I20260202 15:04:27.543677 11421 raft_consensus.cc:2243] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:27.543982 11421 raft_consensus.cc:2272] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:27.545358 11420 ts_tablet_manager.cc:1916] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:27.545357 11421 ts_tablet_manager.cc:1916] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:27.547536 11419 tablet_replica.cc:333] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: stopping tablet replica
I20260202 15:04:27.547823 11419 raft_consensus.cc:2243] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:27.547963 11420 ts_tablet_manager.cc:1929] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:27.548038 11420 log.cc:1199] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/d8e936d79a8d44189a25e31da89b204a
I20260202 15:04:27.548305 11419 raft_consensus.cc:2272] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:27.548364 11420 ts_tablet_manager.cc:1950] T d8e936d79a8d44189a25e31da89b204a P 6845808a908d4457bedf735367f92b97: Deleting consensus metadata
I20260202 15:04:27.549240 11421 ts_tablet_manager.cc:1929] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:27.549320 11421 log.cc:1199] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/d8e936d79a8d44189a25e31da89b204a
I20260202 15:04:27.549646 11421 ts_tablet_manager.cc:1950] T d8e936d79a8d44189a25e31da89b204a P aba950577ef745938c0372f0c49667fe: Deleting consensus metadata
I20260202 15:04:27.549866 11302 tablet_service.cc:1505] Processing CreateTablet for tablet cd2617ecef8448fd9fb3e96faa474b1b (DEFAULT_TABLE table=test-table [id=7e132f1e2083440a976f50f29cb09d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.549984 10866 catalog_manager.cc:4985] TS 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011): tablet d8e936d79a8d44189a25e31da89b204a (table test-table [id=c7f3421db9ea4c888f3199e512720f73]) successfully deleted
I20260202 15:04:27.550115 11302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd2617ecef8448fd9fb3e96faa474b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.550186 11018 tablet_service.cc:1505] Processing CreateTablet for tablet cd2617ecef8448fd9fb3e96faa474b1b (DEFAULT_TABLE table=test-table [id=7e132f1e2083440a976f50f29cb09d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.550413 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd2617ecef8448fd9fb3e96faa474b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.550539 11160 tablet_service.cc:1505] Processing CreateTablet for tablet cd2617ecef8448fd9fb3e96faa474b1b (DEFAULT_TABLE table=test-table [id=7e132f1e2083440a976f50f29cb09d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.550814 11160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd2617ecef8448fd9fb3e96faa474b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.552345 11398 tablet_bootstrap.cc:492] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Bootstrap starting.
I20260202 15:04:27.552521 11397 tablet_bootstrap.cc:492] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Bootstrap starting.
I20260202 15:04:27.553419 11398 tablet_bootstrap.cc:654] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.553632 11397 tablet_bootstrap.cc:654] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.553962 10865 catalog_manager.cc:4985] TS aba950577ef745938c0372f0c49667fe (127.2.164.65:38313): tablet d8e936d79a8d44189a25e31da89b204a (table test-table [id=c7f3421db9ea4c888f3199e512720f73]) successfully deleted
I20260202 15:04:27.554291 11396 tablet_bootstrap.cc:492] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Bootstrap starting.
I20260202 15:04:27.555240 11398 tablet_bootstrap.cc:492] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: No bootstrap required, opened a new log
I20260202 15:04:27.555238 11419 ts_tablet_manager.cc:1916] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:27.555335 11398 ts_tablet_manager.cc:1403] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:27.555322 11396 tablet_bootstrap.cc:654] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.555868 11398 raft_consensus.cc:359] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.555991 11398 raft_consensus.cc:385] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.556028 11398 raft_consensus.cc:740] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6845808a908d4457bedf735367f92b97, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.556209 11398 consensus_queue.cc:260] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.556551 11398 ts_tablet_manager.cc:1434] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:27.556586 11396 tablet_bootstrap.cc:492] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: No bootstrap required, opened a new log
I20260202 15:04:27.556661 11396 ts_tablet_manager.cc:1403] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:27.557164 11396 raft_consensus.cc:359] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.557281 11396 raft_consensus.cc:385] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.557318 11396 raft_consensus.cc:740] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.557433 11396 consensus_queue.cc:260] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.557674 11396 ts_tablet_manager.cc:1434] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:27.559350 11419 ts_tablet_manager.cc:1929] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:27.559492 11419 log.cc:1199] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/d8e936d79a8d44189a25e31da89b204a
I20260202 15:04:27.560002 11419 ts_tablet_manager.cc:1950] T d8e936d79a8d44189a25e31da89b204a P ccb7b766b7c04da3bbaff1ad468894ff: Deleting consensus metadata
I20260202 15:04:27.561116 10864 catalog_manager.cc:4985] TS ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097): tablet d8e936d79a8d44189a25e31da89b204a (table test-table [id=c7f3421db9ea4c888f3199e512720f73]) successfully deleted
I20260202 15:04:27.561915 11397 tablet_bootstrap.cc:492] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: No bootstrap required, opened a new log
I20260202 15:04:27.562007 11397 ts_tablet_manager.cc:1403] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260202 15:04:27.563714 11397 raft_consensus.cc:359] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.563820 11397 raft_consensus.cc:385] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.563843 11397 raft_consensus.cc:740] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba950577ef745938c0372f0c49667fe, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.564002 11397 consensus_queue.cc:260] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.564272 11397 ts_tablet_manager.cc:1434] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260202 15:04:27.614495 11404 raft_consensus.cc:493] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:27.614670 11404 raft_consensus.cc:515] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.616004 11404 leader_election.cc:290] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097)
I20260202 15:04:27.619395 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.616248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36097 (local address 127.2.164.67:42101)
0202 15:04:27.616417 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.616433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.616537 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.616901 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.616905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.616926 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.617220 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.617228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.618089 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.618093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.618983 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.618993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.619117 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.619139 (+ 22us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.619200 (+ 61us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.619259 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260202 15:04:27.619899 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.616504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:42101 (local address 127.2.164.66:36097)
0202 15:04:27.616630 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.616634 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.616648 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:27.616709 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.616715 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.616774 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.616868 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.617350 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.617966 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.619125 (+ 1159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.619611 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.619651 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.619705 (+ 54us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.619768 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260202 15:04:27.620223 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.616248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38313 (local address 127.2.164.67:51765)
0202 15:04:27.616968 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.616983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.617138 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.617458 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.617462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.617481 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:27.617767 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.617789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.618563 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.618567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.619197 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.619206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.619880 (+ 674us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.619896 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.619984 (+ 88us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.620067 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":351,"thread_start_us":95,"threads_started":1}
I20260202 15:04:27.620352 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.616451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:51765 (local address 127.2.164.65:38313)
0202 15:04:27.616586 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.616590 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.617110 (+ 520us) server_negotiation.cc:408] Connection header received
0202 15:04:27.617271 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.617274 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.617351 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.617434 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:27.617920 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.618445 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.619344 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.619770 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.619803 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.620153 (+ 350us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.620239 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260202 15:04:27.620589 11180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd2617ecef8448fd9fb3e96faa474b1b" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" is_pre_election: true
I20260202 15:04:27.620683 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd2617ecef8448fd9fb3e96faa474b1b" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe" is_pre_election: true
I20260202 15:04:27.620805 11038 raft_consensus.cc:2468] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 0.
I20260202 15:04:27.620824 11180 raft_consensus.cc:2468] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 0.
I20260202 15:04:27.621184 11255 leader_election.cc:304] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, aba950577ef745938c0372f0c49667fe; no voters:
I20260202 15:04:27.621469 11404 raft_consensus.cc:2804] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:27.621533 11404 raft_consensus.cc:493] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260202 15:04:27.621553 11369 tablet.cc:2378] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:04:27.621572 11404 raft_consensus.cc:3060] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.622598 11404 raft_consensus.cc:515] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.623034 11404 leader_election.cc:290] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 election: Requested vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097)
I20260202 15:04:27.623353 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd2617ecef8448fd9fb3e96faa474b1b" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe"
I20260202 15:04:27.623452 11180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd2617ecef8448fd9fb3e96faa474b1b" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb7b766b7c04da3bbaff1ad468894ff"
I20260202 15:04:27.623488 11038 raft_consensus.cc:3060] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.623597 11180 raft_consensus.cc:3060] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Advancing to term 1
W20260202 15:04:27.624547 11227 tablet.cc:2378] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:04:27.624706 11180 raft_consensus.cc:2468] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 1.
I20260202 15:04:27.624713 11038 raft_consensus.cc:2468] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 1.
I20260202 15:04:27.625027 11254 leader_election.cc:304] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:27.625222 11404 raft_consensus.cc:2804] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:27.625527 11404 raft_consensus.cc:697] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 1 LEADER]: Becoming Leader. State: Replica: 6845808a908d4457bedf735367f92b97, State: Running, Role: LEADER
I20260202 15:04:27.625969 11404 consensus_queue.cc:237] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.628425 10879 catalog_manager.cc:5654] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6845808a908d4457bedf735367f92b97 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } health_report { overall_health: HEALTHY } } }
W20260202 15:04:27.644030 11085 tablet.cc:2378] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044666, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:27.680320 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.670982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41155 (local address 127.0.0.1:51508)
0202 15:04:27.671307 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.671328 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.671471 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.671761 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.671765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.672329 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:27.672603 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.672613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.673551 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.673556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.674127 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.674138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.674284 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.674862 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:27.674976 (+ 114us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:27.676771 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:27.678660 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.678669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.678674 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.678956 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.679243 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:27.679246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:27.679248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:27.679310 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:27.679691 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:27.679699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:27.679821 (+ 122us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.679949 (+ 128us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.680124 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260202 15:04:27.680356 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.671052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.2.164.126:41155)
0202 15:04:27.671207 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.671219 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.671446 (+ 227us) server_negotiation.cc:408] Connection header received
0202 15:04:27.671582 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.671585 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.671627 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.671703 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.672757 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.673422 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.674270 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.674401 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.676939 (+ 2538us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.676959 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.676962 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.676987 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.678530 (+ 1543us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.679083 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.679086 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.679087 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.679131 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.679405 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.679407 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.679408 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.679577 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.679655 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.679996 (+ 341us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.680239 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":70}
I20260202 15:04:27.684432 10879 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:51508:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260202 15:04:27.685279 10879 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:27.690868 11018 tablet_service.cc:1505] Processing CreateTablet for tablet f5fc7644f3b44cddb43b9b03f14904c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7be60a9c0b7b4ec1b744799de59383b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:04:27.691118 11160 tablet_service.cc:1505] Processing CreateTablet for tablet f5fc7644f3b44cddb43b9b03f14904c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7be60a9c0b7b4ec1b744799de59383b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:04:27.691329 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5fc7644f3b44cddb43b9b03f14904c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.691452 11160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5fc7644f3b44cddb43b9b03f14904c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.694356 11396 tablet_bootstrap.cc:492] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: Bootstrap starting.
I20260202 15:04:27.694527 11397 tablet_bootstrap.cc:492] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: Bootstrap starting.
I20260202 15:04:27.694914 11302 tablet_service.cc:1505] Processing CreateTablet for tablet f5fc7644f3b44cddb43b9b03f14904c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7be60a9c0b7b4ec1b744799de59383b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260202 15:04:27.695312 11302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5fc7644f3b44cddb43b9b03f14904c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.695513 11396 tablet_bootstrap.cc:654] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.695586 11397 tablet_bootstrap.cc:654] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.697568 11397 tablet_bootstrap.cc:492] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: No bootstrap required, opened a new log
I20260202 15:04:27.697677 11397 ts_tablet_manager.cc:1403] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260202 15:04:27.698216 11397 raft_consensus.cc:359] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.698339 11397 raft_consensus.cc:385] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.698374 11397 raft_consensus.cc:740] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba950577ef745938c0372f0c49667fe, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.698477 11398 tablet_bootstrap.cc:492] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: Bootstrap starting.
I20260202 15:04:27.698514 11397 consensus_queue.cc:260] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.698796 11403 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.698925 11403 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.698810 11397 ts_tablet_manager.cc:1434] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260202 15:04:27.699594 11398 tablet_bootstrap.cc:654] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.700502 11396 tablet_bootstrap.cc:492] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: No bootstrap required, opened a new log
I20260202 15:04:27.700613 11396 ts_tablet_manager.cc:1403] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260202 15:04:27.701236 11396 raft_consensus.cc:359] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.701375 11396 raft_consensus.cc:385] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.701416 11396 raft_consensus.cc:740] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.701542 11398 tablet_bootstrap.cc:492] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: No bootstrap required, opened a new log
I20260202 15:04:27.701550 11396 consensus_queue.cc:260] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.701638 11398 ts_tablet_manager.cc:1403] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:27.702112 11398 raft_consensus.cc:359] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.702250 11398 raft_consensus.cc:385] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.702286 11398 raft_consensus.cc:740] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6845808a908d4457bedf735367f92b97, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.702406 11398 consensus_queue.cc:260] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.702661 11398 ts_tablet_manager.cc:1434] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:27.702709 11404 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.702843 11404 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.703495 11406 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.703644 11406 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.704397 11396 ts_tablet_manager.cc:1434] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260202 15:04:27.796149 11404 raft_consensus.cc:493] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:27.796321 11404 raft_consensus.cc:515] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.796815 11404 leader_election.cc:290] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097)
I20260202 15:04:27.797384 11180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5fc7644f3b44cddb43b9b03f14904c7" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" is_pre_election: true
I20260202 15:04:27.797379 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5fc7644f3b44cddb43b9b03f14904c7" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe" is_pre_election: true
I20260202 15:04:27.797516 11180 raft_consensus.cc:2468] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 0.
I20260202 15:04:27.797565 11038 raft_consensus.cc:2468] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 0.
I20260202 15:04:27.797917 11254 leader_election.cc:304] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:27.798117 11404 raft_consensus.cc:2804] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:27.798202 11404 raft_consensus.cc:493] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:27.798264 11404 raft_consensus.cc:3060] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.799245 11404 raft_consensus.cc:515] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.799742 11404 leader_election.cc:290] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 election: Requested vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097)
I20260202 15:04:27.800102 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5fc7644f3b44cddb43b9b03f14904c7" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe"
I20260202 15:04:27.800112 11180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5fc7644f3b44cddb43b9b03f14904c7" candidate_uuid: "6845808a908d4457bedf735367f92b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb7b766b7c04da3bbaff1ad468894ff"
I20260202 15:04:27.800244 11038 raft_consensus.cc:3060] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.800257 11180 raft_consensus.cc:3060] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:27.801136 11038 raft_consensus.cc:2468] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 1.
I20260202 15:04:27.801168 11180 raft_consensus.cc:2468] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6845808a908d4457bedf735367f92b97 in term 1.
I20260202 15:04:27.801456 11254 leader_election.cc:304] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:27.801635 11404 raft_consensus.cc:2804] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:27.801774 11404 raft_consensus.cc:697] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [term 1 LEADER]: Becoming Leader. State: Replica: 6845808a908d4457bedf735367f92b97, State: Running, Role: LEADER
I20260202 15:04:27.801939 11404 consensus_queue.cc:237] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.803032 11426 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: TxnStatusTablet state changed. Reason: New leader 6845808a908d4457bedf735367f92b97. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.803138 11426 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:04:27.803561 11438 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260202 15:04:27.803725 11438 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260202 15:04:27.803988 10879 catalog_manager.cc:5654] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6845808a908d4457bedf735367f92b97 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } health_report { overall_health: HEALTHY } } }
I20260202 15:04:27.833108 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.829777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.2.164.67:42011)
0202 15:04:27.829927 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.829932 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.830037 (+ 105us) server_negotiation.cc:408] Connection header received
0202 15:04:27.830274 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.830278 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.830336 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.830434 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:27.830793 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.831304 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.831955 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.832157 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.832208 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:27.832668 (+ 460us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:27.832793 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.832883 (+ 90us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.832933 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:04:27.833108 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.829684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42011 (local address 127.0.0.1:55334)
0202 15:04:27.829942 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.829960 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.830119 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.830456 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.830459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.830468 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:04:27.830679 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.830685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.831421 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.831426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.831838 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.831845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.831972 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.832023 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:04:27.832770 (+ 747us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:04:27.832786 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.832862 (+ 76us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.832939 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260202 15:04:27.836094 11038 raft_consensus.cc:1275] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Refusing update from remote peer 6845808a908d4457bedf735367f92b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:27.836162 11180 raft_consensus.cc:1275] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Refusing update from remote peer 6845808a908d4457bedf735367f92b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:27.836704 11426 consensus_queue.cc:1048] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:27.836889 11404 consensus_queue.cc:1048] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:27.838721 11403 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: TxnStatusTablet state changed. Reason: New leader 6845808a908d4457bedf735367f92b97. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.838846 11403 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.840008 11406 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: TxnStatusTablet state changed. Reason: New leader 6845808a908d4457bedf735367f92b97. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.840131 11406 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.841661 11426 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.841770 11426 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:04:27.841869 11411 mvcc.cc:204] Tried to move back new op lower bound from 7250102959451037696 to 7250102959318106112. Current Snapshot: MvccSnapshot[applied={T|T < 7250102959451037696}]
I20260202 15:04:27.843199 11403 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.843299 11403 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P aba950577ef745938c0372f0c49667fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.844678 11404 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.844774 11404 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P 6845808a908d4457bedf735367f92b97: This TxnStatusTablet replica's current role is: LEADER
I20260202 15:04:27.845892 11406 tablet_replica.cc:442] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6845808a908d4457bedf735367f92b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } } }
I20260202 15:04:27.846009 11406 tablet_replica.cc:445] T f5fc7644f3b44cddb43b9b03f14904c7 P ccb7b766b7c04da3bbaff1ad468894ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260202 15:04:27.850311 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.847199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.2.164.67:42011)
0202 15:04:27.847328 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.847332 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.847503 (+ 171us) server_negotiation.cc:408] Connection header received
0202 15:04:27.847640 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.847643 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.847821 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.847921 (+ 100us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:27.848347 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.848814 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.849608 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.849792 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.849834 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:27.849901 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:27.849993 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.850116 (+ 123us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.850166 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:04:27.860813 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.857890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:53371 (local address 127.2.164.67:42011)
0202 15:04:27.858013 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.858037 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.858103 (+ 66us) server_negotiation.cc:408] Connection header received
0202 15:04:27.858216 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.858220 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.858272 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.858359 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:27.858675 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.859427 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.860116 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.860302 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.860346 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:27.860424 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:27.860502 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.860609 (+ 107us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.860655 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:04:27.860826 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.857854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42011 (local address 127.2.164.67:53371)
0202 15:04:27.858019 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:27.858032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:27.858126 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:27.858372 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:27.858374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:27.858381 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:04:27.858577 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.858582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.859561 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.859564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:27.860008 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:27.860015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.860107 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.860145 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:04:27.860516 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:04:27.860520 (+ 4us) client_negotiation.cc:769] Sending connection context
0202 15:04:27.860576 (+ 56us) client_negotiation.cc:240] Negotiation successful
0202 15:04:27.860628 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260202 15:04:27.868019 11180 raft_consensus.cc:1275] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Refusing update from remote peer 6845808a908d4457bedf735367f92b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:27.868165 11038 raft_consensus.cc:1275] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Refusing update from remote peer 6845808a908d4457bedf735367f92b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:27.868487 11404 consensus_queue.cc:1048] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:27.868629 11445 consensus_queue.cc:1048] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:27.924784 10879 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:27.924928 10879 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:27.927340 10879 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Sending DeleteTablet for 3 replicas of tablet cd2617ecef8448fd9fb3e96faa474b1b
I20260202 15:04:27.927944 11018 tablet_service.cc:1552] Processing DeleteTablet for tablet cd2617ecef8448fd9fb3e96faa474b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:27 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:59862
I20260202 15:04:27.927964 11160 tablet_service.cc:1552] Processing DeleteTablet for tablet cd2617ecef8448fd9fb3e96faa474b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:27 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:41140
I20260202 15:04:27.928174 11419 tablet_replica.cc:333] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: stopping tablet replica
I20260202 15:04:27.928308 11419 raft_consensus.cc:2243] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:27.928486 11419 raft_consensus.cc:2272] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:27.928596 11302 tablet_service.cc:1552] Processing DeleteTablet for tablet cd2617ecef8448fd9fb3e96faa474b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:27 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260202 15:04:27.928694 11421 tablet_replica.cc:333] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: stopping tablet replica
I20260202 15:04:27.928813 11421 raft_consensus.cc:2243] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:27.928864 11420 tablet_replica.cc:333] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: stopping tablet replica
I20260202 15:04:27.928934 11421 raft_consensus.cc:2272] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:27.928966 11420 raft_consensus.cc:2243] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:27.929322 11420 raft_consensus.cc:2272] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:27.929687 11419 ts_tablet_manager.cc:1916] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:27.929910 11421 ts_tablet_manager.cc:1916] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:27.932333 11419 ts_tablet_manager.cc:1929] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:04:27.932416 11419 log.cc:1199] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/cd2617ecef8448fd9fb3e96faa474b1b
I20260202 15:04:27.932677 11419 ts_tablet_manager.cc:1950] T cd2617ecef8448fd9fb3e96faa474b1b P ccb7b766b7c04da3bbaff1ad468894ff: Deleting consensus metadata
I20260202 15:04:27.933257 11421 ts_tablet_manager.cc:1929] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:04:27.933367 11421 log.cc:1199] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/cd2617ecef8448fd9fb3e96faa474b1b
I20260202 15:04:27.933504 10864 catalog_manager.cc:4985] TS ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097): tablet cd2617ecef8448fd9fb3e96faa474b1b (table test-table [id=7e132f1e2083440a976f50f29cb09d70]) successfully deleted
I20260202 15:04:27.933627 11421 ts_tablet_manager.cc:1950] T cd2617ecef8448fd9fb3e96faa474b1b P aba950577ef745938c0372f0c49667fe: Deleting consensus metadata
I20260202 15:04:27.934194 11420 ts_tablet_manager.cc:1916] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:27.934392 10865 catalog_manager.cc:4985] TS aba950577ef745938c0372f0c49667fe (127.2.164.65:38313): tablet cd2617ecef8448fd9fb3e96faa474b1b (table test-table [id=7e132f1e2083440a976f50f29cb09d70]) successfully deleted
I20260202 15:04:27.936688 11420 ts_tablet_manager.cc:1929] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:04:27.936753 11420 log.cc:1199] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/cd2617ecef8448fd9fb3e96faa474b1b
I20260202 15:04:27.937337 11420 ts_tablet_manager.cc:1950] T cd2617ecef8448fd9fb3e96faa474b1b P 6845808a908d4457bedf735367f92b97: Deleting consensus metadata
I20260202 15:04:27.938367 10866 catalog_manager.cc:4985] TS 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011): tablet cd2617ecef8448fd9fb3e96faa474b1b (table test-table [id=7e132f1e2083440a976f50f29cb09d70]) successfully deleted
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260202 15:04:27.957177 10928 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:27.950639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.2.164.126:41155)
0202 15:04:27.950805 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.950810 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.950975 (+ 165us) server_negotiation.cc:408] Connection header received
0202 15:04:27.951118 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.951120 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.951181 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.951248 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.952291 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.952789 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.953734 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.953883 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.956991 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51518: BlockingRecv error: recv got EOF from 127.0.0.1:51518 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Feb 02 15:04:27 dist-test-slave-6zzp krb5kdc[10831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260202 15:04:27.966475 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:27.958794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.2.164.126:41155)
0202 15:04:27.958941 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:27.958946 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:27.959026 (+ 80us) server_negotiation.cc:408] Connection header received
0202 15:04:27.959135 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:27.959138 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:27.959204 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:27.959298 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:27.959939 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.960717 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:27.961404 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:27.961596 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:27.963508 (+ 1912us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:27.963526 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:27.963530 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:27.963558 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:27.965080 (+ 1522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.965432 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.965438 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.965443 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.965494 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:27.965700 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:27.965703 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:27.965705 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:27.965885 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:27.966010 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:27.966210 (+ 200us) server_negotiation.cc:300] Negotiation successful
0202 15:04:27.966324 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260202 15:04:27.968943 10879 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:04:27.969467 10879 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:27.974468 11018 tablet_service.cc:1505] Processing CreateTablet for tablet 8f8fb744a6714eaea32a30eb70b1523d (DEFAULT_TABLE table=test-table [id=edd623e89ad04dd68fd923d0d0283973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.974778 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8fb744a6714eaea32a30eb70b1523d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.974786 11302 tablet_service.cc:1505] Processing CreateTablet for tablet 8f8fb744a6714eaea32a30eb70b1523d (DEFAULT_TABLE table=test-table [id=edd623e89ad04dd68fd923d0d0283973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.974910 11160 tablet_service.cc:1505] Processing CreateTablet for tablet 8f8fb744a6714eaea32a30eb70b1523d (DEFAULT_TABLE table=test-table [id=edd623e89ad04dd68fd923d0d0283973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:27.975071 11302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8fb744a6714eaea32a30eb70b1523d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.975159 11160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8fb744a6714eaea32a30eb70b1523d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:27.977144 11397 tablet_bootstrap.cc:492] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Bootstrap starting.
I20260202 15:04:27.977344 11398 tablet_bootstrap.cc:492] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Bootstrap starting.
I20260202 15:04:27.978258 11397 tablet_bootstrap.cc:654] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.978328 11398 tablet_bootstrap.cc:654] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.979574 11396 tablet_bootstrap.cc:492] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Bootstrap starting.
I20260202 15:04:27.979847 11398 tablet_bootstrap.cc:492] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: No bootstrap required, opened a new log
I20260202 15:04:27.979926 11398 ts_tablet_manager.cc:1403] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:27.980301 11396 tablet_bootstrap.cc:654] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:27.980422 11398 raft_consensus.cc:359] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.980537 11398 raft_consensus.cc:385] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.980568 11398 raft_consensus.cc:740] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6845808a908d4457bedf735367f92b97, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.980700 11398 consensus_queue.cc:260] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.980962 11398 ts_tablet_manager.cc:1434] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:27.981415 11397 tablet_bootstrap.cc:492] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: No bootstrap required, opened a new log
I20260202 15:04:27.981510 11397 ts_tablet_manager.cc:1403] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260202 15:04:27.982000 11397 raft_consensus.cc:359] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.982085 11397 raft_consensus.cc:385] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.982106 11397 raft_consensus.cc:740] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba950577ef745938c0372f0c49667fe, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.982203 11397 consensus_queue.cc:260] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.982440 11397 ts_tablet_manager.cc:1434] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260202 15:04:27.983511 11396 tablet_bootstrap.cc:492] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: No bootstrap required, opened a new log
I20260202 15:04:27.983609 11396 ts_tablet_manager.cc:1403] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260202 15:04:27.984120 11396 raft_consensus.cc:359] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.984289 11396 raft_consensus.cc:385] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:27.984334 11396 raft_consensus.cc:740] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Initialized, Role: FOLLOWER
I20260202 15:04:27.984477 11396 consensus_queue.cc:260] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:27.984750 11396 ts_tablet_manager.cc:1434] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260202 15:04:28.186630 11406 raft_consensus.cc:493] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:28.186779 11406 raft_consensus.cc:515] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.187256 11406 leader_election.cc:290] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:28.187737 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f8fb744a6714eaea32a30eb70b1523d" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe" is_pre_election: true
I20260202 15:04:28.187753 11320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f8fb744a6714eaea32a30eb70b1523d" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6845808a908d4457bedf735367f92b97" is_pre_election: true
I20260202 15:04:28.187894 11038 raft_consensus.cc:2468] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 0.
I20260202 15:04:28.187906 11320 raft_consensus.cc:2468] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 0.
I20260202 15:04:28.188249 11113 leader_election.cc:304] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba950577ef745938c0372f0c49667fe, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:28.188444 11406 raft_consensus.cc:2804] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:28.188517 11406 raft_consensus.cc:493] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:28.188572 11406 raft_consensus.cc:3060] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:28.189615 11406 raft_consensus.cc:515] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.189991 11406 leader_election.cc:290] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 election: Requested vote from peers aba950577ef745938c0372f0c49667fe (127.2.164.65:38313), 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:28.190335 11320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f8fb744a6714eaea32a30eb70b1523d" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6845808a908d4457bedf735367f92b97"
I20260202 15:04:28.190388 11038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f8fb744a6714eaea32a30eb70b1523d" candidate_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba950577ef745938c0372f0c49667fe"
I20260202 15:04:28.190460 11320 raft_consensus.cc:3060] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:28.190490 11038 raft_consensus.cc:3060] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:28.191488 11320 raft_consensus.cc:2468] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 1.
I20260202 15:04:28.191488 11038 raft_consensus.cc:2468] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb7b766b7c04da3bbaff1ad468894ff in term 1.
I20260202 15:04:28.191768 11114 leader_election.cc:304] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, ccb7b766b7c04da3bbaff1ad468894ff; no voters:
I20260202 15:04:28.191998 11406 raft_consensus.cc:2804] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:28.192191 11406 raft_consensus.cc:697] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 1 LEADER]: Becoming Leader. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Running, Role: LEADER
I20260202 15:04:28.192348 11406 consensus_queue.cc:237] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.194255 10878 catalog_manager.cc:5654] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } health_report { overall_health: UNKNOWN } } }
I20260202 15:04:28.223148 11467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.220549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.2.164.66:36097)
0202 15:04:28.220766 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:28.220771 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:28.220783 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:28.220828 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:28.220831 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:28.220877 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:28.220957 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:28.221303 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.221779 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.222469 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.222598 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.222634 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:28.222700 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:28.222761 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:28.222897 (+ 136us) server_negotiation.cc:300] Negotiation successful
0202 15:04:28.222941 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260202 15:04:28.225178 11038 raft_consensus.cc:1275] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Refusing update from remote peer ccb7b766b7c04da3bbaff1ad468894ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:28.225221 11320 raft_consensus.cc:1275] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Refusing update from remote peer ccb7b766b7c04da3bbaff1ad468894ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:28.225611 11406 consensus_queue.cc:1048] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:28.225812 11464 consensus_queue.cc:1048] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:28.232870 10879 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:28.233038 10879 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:28.235396 10879 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Sending DeleteTablet for 3 replicas of tablet 8f8fb744a6714eaea32a30eb70b1523d
I20260202 15:04:28.236003 11018 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f8fb744a6714eaea32a30eb70b1523d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:28 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:59862
I20260202 15:04:28.236016 11160 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f8fb744a6714eaea32a30eb70b1523d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:28 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:41140
I20260202 15:04:28.236174 11421 tablet_replica.cc:333] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: stopping tablet replica
I20260202 15:04:28.236202 11302 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f8fb744a6714eaea32a30eb70b1523d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:28 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260202 15:04:28.236243 11419 tablet_replica.cc:333] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: stopping tablet replica
I20260202 15:04:28.236287 11421 raft_consensus.cc:2243] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:28.236380 11419 raft_consensus.cc:2243] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:28.236394 11420 tablet_replica.cc:333] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: stopping tablet replica
I20260202 15:04:28.236407 11421 raft_consensus.cc:2272] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:28.236488 11420 raft_consensus.cc:2243] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:28.236562 11419 raft_consensus.cc:2272] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:28.236586 11420 raft_consensus.cc:2272] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:28.236912 10879 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:04:28.237354 10879 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:28.237370 11421 ts_tablet_manager.cc:1916] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:28.237381 11419 ts_tablet_manager.cc:1916] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:28.237586 11420 ts_tablet_manager.cc:1916] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:28.239682 11419 ts_tablet_manager.cc:1929] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:28.239687 11420 ts_tablet_manager.cc:1929] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:28.239756 11420 log.cc:1199] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/8f8fb744a6714eaea32a30eb70b1523d
I20260202 15:04:28.239756 11419 log.cc:1199] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/8f8fb744a6714eaea32a30eb70b1523d
I20260202 15:04:28.239837 11421 ts_tablet_manager.cc:1929] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:28.239903 11421 log.cc:1199] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-0/wal/wals/8f8fb744a6714eaea32a30eb70b1523d
I20260202 15:04:28.240025 11420 ts_tablet_manager.cc:1950] T 8f8fb744a6714eaea32a30eb70b1523d P 6845808a908d4457bedf735367f92b97: Deleting consensus metadata
I20260202 15:04:28.240027 11419 ts_tablet_manager.cc:1950] T 8f8fb744a6714eaea32a30eb70b1523d P ccb7b766b7c04da3bbaff1ad468894ff: Deleting consensus metadata
I20260202 15:04:28.240690 10866 catalog_manager.cc:4985] TS 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011): tablet 8f8fb744a6714eaea32a30eb70b1523d (table test-table [id=edd623e89ad04dd68fd923d0d0283973]) successfully deleted
I20260202 15:04:28.240849 10864 catalog_manager.cc:4985] TS ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097): tablet 8f8fb744a6714eaea32a30eb70b1523d (table test-table [id=edd623e89ad04dd68fd923d0d0283973]) successfully deleted
I20260202 15:04:28.241681 11421 ts_tablet_manager.cc:1950] T 8f8fb744a6714eaea32a30eb70b1523d P aba950577ef745938c0372f0c49667fe: Deleting consensus metadata
I20260202 15:04:28.242432 10865 catalog_manager.cc:4985] TS aba950577ef745938c0372f0c49667fe (127.2.164.65:38313): tablet 8f8fb744a6714eaea32a30eb70b1523d (table test-table [id=edd623e89ad04dd68fd923d0d0283973]) successfully deleted
I20260202 15:04:28.243338 11018 tablet_service.cc:1505] Processing CreateTablet for tablet 0f7e769268c54fdaae6773bb44549c36 (DEFAULT_TABLE table=test-table [id=c9dd26dd94b84d99904f411acd303729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:28.243355 11160 tablet_service.cc:1505] Processing CreateTablet for tablet 0f7e769268c54fdaae6773bb44549c36 (DEFAULT_TABLE table=test-table [id=c9dd26dd94b84d99904f411acd303729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:28.243481 11302 tablet_service.cc:1505] Processing CreateTablet for tablet 0f7e769268c54fdaae6773bb44549c36 (DEFAULT_TABLE table=test-table [id=c9dd26dd94b84d99904f411acd303729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:28.243609 11160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7e769268c54fdaae6773bb44549c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:28.243630 11018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7e769268c54fdaae6773bb44549c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:28.243708 11302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7e769268c54fdaae6773bb44549c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:28.245407 11397 tablet_bootstrap.cc:492] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: Bootstrap starting.
I20260202 15:04:28.245750 11398 tablet_bootstrap.cc:492] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Bootstrap starting.
I20260202 15:04:28.246454 11397 tablet_bootstrap.cc:654] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:28.246459 11398 tablet_bootstrap.cc:654] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:28.247051 11396 tablet_bootstrap.cc:492] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Bootstrap starting.
I20260202 15:04:28.247539 11398 tablet_bootstrap.cc:492] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: No bootstrap required, opened a new log
I20260202 15:04:28.247622 11397 tablet_bootstrap.cc:492] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: No bootstrap required, opened a new log
I20260202 15:04:28.247628 11398 ts_tablet_manager.cc:1403] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260202 15:04:28.247699 11397 ts_tablet_manager.cc:1403] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:28.247861 11396 tablet_bootstrap.cc:654] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:28.248034 11398 raft_consensus.cc:359] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.248157 11398 raft_consensus.cc:385] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:28.248193 11398 raft_consensus.cc:740] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6845808a908d4457bedf735367f92b97, State: Initialized, Role: FOLLOWER
I20260202 15:04:28.248152 11397 raft_consensus.cc:359] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.248250 11397 raft_consensus.cc:385] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:28.248288 11397 raft_consensus.cc:740] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba950577ef745938c0372f0c49667fe, State: Initialized, Role: FOLLOWER
I20260202 15:04:28.248338 11398 consensus_queue.cc:260] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.248402 11397 consensus_queue.cc:260] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.248597 11398 ts_tablet_manager.cc:1434] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:28.248644 11397 ts_tablet_manager.cc:1434] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:28.249819 11396 tablet_bootstrap.cc:492] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: No bootstrap required, opened a new log
I20260202 15:04:28.249919 11396 ts_tablet_manager.cc:1403] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260202 15:04:28.250336 11396 raft_consensus.cc:359] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.250463 11396 raft_consensus.cc:385] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:28.250512 11396 raft_consensus.cc:740] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb7b766b7c04da3bbaff1ad468894ff, State: Initialized, Role: FOLLOWER
I20260202 15:04:28.250638 11396 consensus_queue.cc:260] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.250902 11396 ts_tablet_manager.cc:1434] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260202 15:04:28.363859 11403 raft_consensus.cc:493] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:28.364045 11403 raft_consensus.cc:515] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.365343 11403 leader_election.cc:290] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097), 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:28.368512 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.365553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42011 (local address 127.2.164.65:50933)
0202 15:04:28.365872 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:28.365887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:28.365999 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:28.366294 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:28.366297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:28.366315 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:28.366554 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.366560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.367373 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.367379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:28.367979 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.367985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.368114 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.368136 (+ 22us) client_negotiation.cc:769] Sending connection context
0202 15:04:28.368285 (+ 149us) client_negotiation.cc:240] Negotiation successful
0202 15:04:28.368346 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":68,"threads_started":1}
I20260202 15:04:28.368516 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.365579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36097 (local address 127.2.164.65:49325)
0202 15:04:28.366009 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:28.366025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:28.366175 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:28.366469 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:28.366473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:28.366493 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:28.366734 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.366748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.367486 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.367492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:28.368077 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.368085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.368223 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.368242 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:04:28.368298 (+ 56us) client_negotiation.cc:240] Negotiation successful
0202 15:04:28.368355 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":85,"threads_started":1}
I20260202 15:04:28.368873 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.365705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50933 (local address 127.2.164.67:42011)
0202 15:04:28.365943 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:28.365947 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:28.365963 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:04:28.366130 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:28.366133 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:28.366180 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:28.366271 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:28.366683 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.367262 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.368099 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.368572 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.368615 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:28.368666 (+ 51us) server_negotiation.cc:300] Negotiation successful
0202 15:04:28.368732 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
I20260202 15:04:28.369279 11320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f7e769268c54fdaae6773bb44549c36" candidate_uuid: "aba950577ef745938c0372f0c49667fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6845808a908d4457bedf735367f92b97" is_pre_election: true
I20260202 15:04:28.369359 11467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.365681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:49325 (local address 127.2.164.66:36097)
0202 15:04:28.365838 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:28.365846 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:28.366107 (+ 261us) server_negotiation.cc:408] Connection header received
0202 15:04:28.366298 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:28.366303 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:28.366365 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:28.366459 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:28.366869 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.367363 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.368584 (+ 1221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.369072 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.369111 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:28.369165 (+ 54us) server_negotiation.cc:300] Negotiation successful
0202 15:04:28.369231 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260202 15:04:28.369443 11320 raft_consensus.cc:2468] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba950577ef745938c0372f0c49667fe in term 0.
I20260202 15:04:28.369798 11179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f7e769268c54fdaae6773bb44549c36" candidate_uuid: "aba950577ef745938c0372f0c49667fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" is_pre_election: true
I20260202 15:04:28.369966 11179 raft_consensus.cc:2468] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba950577ef745938c0372f0c49667fe in term 0.
I20260202 15:04:28.369946 10972 leader_election.cc:304] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, aba950577ef745938c0372f0c49667fe; no voters:
I20260202 15:04:28.370211 11403 raft_consensus.cc:2804] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:28.370277 11403 raft_consensus.cc:493] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:28.370318 11403 raft_consensus.cc:3060] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:28.371477 11403 raft_consensus.cc:515] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.371902 11403 leader_election.cc:290] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [CANDIDATE]: Term 1 election: Requested vote from peers ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097), 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011)
I20260202 15:04:28.372265 11320 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f7e769268c54fdaae6773bb44549c36" candidate_uuid: "aba950577ef745938c0372f0c49667fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6845808a908d4457bedf735367f92b97"
I20260202 15:04:28.372316 11179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f7e769268c54fdaae6773bb44549c36" candidate_uuid: "aba950577ef745938c0372f0c49667fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb7b766b7c04da3bbaff1ad468894ff"
I20260202 15:04:28.372370 11320 raft_consensus.cc:3060] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:28.372426 11179 raft_consensus.cc:3060] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:28.373384 11320 raft_consensus.cc:2468] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba950577ef745938c0372f0c49667fe in term 1.
I20260202 15:04:28.373560 11179 raft_consensus.cc:2468] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba950577ef745938c0372f0c49667fe in term 1.
I20260202 15:04:28.373713 10972 leader_election.cc:304] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6845808a908d4457bedf735367f92b97, aba950577ef745938c0372f0c49667fe; no voters:
I20260202 15:04:28.373910 11403 raft_consensus.cc:2804] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:28.374116 11403 raft_consensus.cc:697] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 1 LEADER]: Becoming Leader. State: Replica: aba950577ef745938c0372f0c49667fe, State: Running, Role: LEADER
I20260202 15:04:28.374383 11403 consensus_queue.cc:237] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } }
I20260202 15:04:28.376685 10877 catalog_manager.cc:5654] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe reported cstate change: term changed from 0 to 1, leader changed from <none> to aba950577ef745938c0372f0c49667fe (127.2.164.65). New cstate: current_term: 1 leader_uuid: "aba950577ef745938c0372f0c49667fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba950577ef745938c0372f0c49667fe" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 } health_report { overall_health: UNKNOWN } } }
I20260202 15:04:28.407456 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.404356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.2.164.65:38313)
0202 15:04:28.404624 (+ 268us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:28.404628 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:28.404639 (+ 11us) server_negotiation.cc:408] Connection header received
0202 15:04:28.404710 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:28.404713 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:28.404764 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:28.404858 (+ 94us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:28.405208 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.405714 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.406371 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.406520 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.406572 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:28.407086 (+ 514us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:28.407189 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:28.407282 (+ 93us) server_negotiation.cc:300] Negotiation successful
0202 15:04:28.407332 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":114,"threads_started":1}
I20260202 15:04:28.417537 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.414241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42011 (local address 127.2.164.65:41803)
0202 15:04:28.414583 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:28.414600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:28.414749 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:28.415082 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:28.415085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:28.415095 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:04:28.415363 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.415370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.416101 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.416107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:28.416562 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.416570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.416728 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.416781 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:04:28.417202 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:04:28.417210 (+ 8us) client_negotiation.cc:769] Sending connection context
0202 15:04:28.417305 (+ 95us) client_negotiation.cc:240] Negotiation successful
0202 15:04:28.417384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
I20260202 15:04:28.417520 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.414424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41803 (local address 127.2.164.67:42011)
0202 15:04:28.414574 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:28.414579 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:28.414686 (+ 107us) server_negotiation.cc:408] Connection header received
0202 15:04:28.414877 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:28.414891 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:28.414970 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:28.415068 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:28.415487 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.415984 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.416687 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.416888 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.416955 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:28.417101 (+ 146us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:28.417191 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:28.417316 (+ 125us) server_negotiation.cc:300] Negotiation successful
0202 15:04:28.417366 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260202 15:04:28.424028 11320 raft_consensus.cc:1275] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Refusing update from remote peer aba950577ef745938c0372f0c49667fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:28.424044 11179 raft_consensus.cc:1275] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Refusing update from remote peer aba950577ef745938c0372f0c49667fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:28.424494 11476 consensus_queue.cc:1048] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb7b766b7c04da3bbaff1ad468894ff" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:28.424708 11403 consensus_queue.cc:1048] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6845808a908d4457bedf735367f92b97" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:28.442631 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.439588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:37695 (local address 127.2.164.65:38313)
0202 15:04:28.439758 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:28.439763 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:28.439884 (+ 121us) server_negotiation.cc:408] Connection header received
0202 15:04:28.439989 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:28.439992 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:28.440039 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:28.440103 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:28.440442 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.440933 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.441710 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.441963 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.442022 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:28.442099 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:28.442181 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:28.442422 (+ 241us) server_negotiation.cc:300] Negotiation successful
0202 15:04:28.442476 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260202 15:04:28.442711 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.439486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38313 (local address 127.2.164.67:37695)
0202 15:04:28.439788 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:28.439805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:28.439915 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:28.440122 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:28.440125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:28.440131 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0202 15:04:28.440348 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.440353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.441087 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:28.441091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:28.441594 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:28.441603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:28.441725 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:28.441756 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0202 15:04:28.442190 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0202 15:04:28.442204 (+ 14us) client_negotiation.cc:769] Sending connection context
0202 15:04:28.442412 (+ 208us) client_negotiation.cc:240] Negotiation successful
0202 15:04:28.442498 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":70,"threads_started":1}
I20260202 15:04:28.466264 10879 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:28.466442 10879 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:28.468616 10879 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 165f322e88af4b839a53e3c7b0bfb936: Sending DeleteTablet for 3 replicas of tablet 0f7e769268c54fdaae6773bb44549c36
I20260202 15:04:28.469264 11018 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f7e769268c54fdaae6773bb44549c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:28 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:59862
I20260202 15:04:28.469282 11160 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f7e769268c54fdaae6773bb44549c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:28 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:41140
I20260202 15:04:28.469355 11302 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f7e769268c54fdaae6773bb44549c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:28 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260202 15:04:28.469453 11421 tablet_replica.cc:333] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: stopping tablet replica
I20260202 15:04:28.469452 11419 tablet_replica.cc:333] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: stopping tablet replica
I20260202 15:04:28.469543 11420 tablet_replica.cc:333] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: stopping tablet replica
I20260202 15:04:28.469563 11419 raft_consensus.cc:2243] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:28.469560 11421 raft_consensus.cc:2243] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:28.469653 11420 raft_consensus.cc:2243] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:28.469681 11419 raft_consensus.cc:2272] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:28.469770 11420 raft_consensus.cc:2272] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:28.469877 11421 raft_consensus.cc:2272] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:28.470443 11419 ts_tablet_manager.cc:1916] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:28.470686 11421 ts_tablet_manager.cc:1916] T 0f7e769268c54fdaae6773bb44549c36 P aba950577ef745938c0372f0c49667fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:28.471050 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 10953
I20260202 15:04:28.472754 11419 ts_tablet_manager.cc:1929] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:04:28.472860 11419 log.cc:1199] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/0f7e769268c54fdaae6773bb44549c36
I20260202 15:04:28.473656 11419 ts_tablet_manager.cc:1950] T 0f7e769268c54fdaae6773bb44549c36 P ccb7b766b7c04da3bbaff1ad468894ff: Deleting consensus metadata
I20260202 15:04:28.473835 11420 ts_tablet_manager.cc:1916] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:28.474371 10864 catalog_manager.cc:4985] TS ccb7b766b7c04da3bbaff1ad468894ff (127.2.164.66:36097): tablet 0f7e769268c54fdaae6773bb44549c36 (table test-table [id=c9dd26dd94b84d99904f411acd303729]) successfully deleted
I20260202 15:04:28.475757 11420 ts_tablet_manager.cc:1929] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260202 15:04:28.475821 11420 log.cc:1199] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/0f7e769268c54fdaae6773bb44549c36
I20260202 15:04:28.476058 11420 ts_tablet_manager.cc:1950] T 0f7e769268c54fdaae6773bb44549c36 P 6845808a908d4457bedf735367f92b97: Deleting consensus metadata
I20260202 15:04:28.476807 10866 catalog_manager.cc:4985] TS 6845808a908d4457bedf735367f92b97 (127.2.164.67:42011): tablet 0f7e769268c54fdaae6773bb44549c36 (table test-table [id=c9dd26dd94b84d99904f411acd303729]) successfully deleted
I20260202 15:04:28.478933 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 11095
W20260202 15:04:28.479050 10865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:38313 (error 108)
I20260202 15:04:28.481354 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:28.480900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38313 (local address 127.0.0.1:59866)
0202 15:04:28.481164 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:28.481264 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:38313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":56,"threads_started":1}
W20260202 15:04:28.481492 10865 catalog_manager.cc:4712] TS aba950577ef745938c0372f0c49667fe (127.2.164.65:38313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f7e769268c54fdaae6773bb44549c36: Network error: Client connection negotiation failed: client connection to 127.2.164.65:38313: connect: Connection refused (error 111)
I20260202 15:04:28.486702 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 11237
I20260202 15:04:28.493634 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 10847
2026-02-02T15:04:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4261 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:28 dist-test-slave-6zzp krb5kdc[11493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:28 dist-test-slave-6zzp krb5kdc[11493](info): set up 2 sockets
Feb 02 15:04:28 dist-test-slave-6zzp krb5kdc[11493](info): commencing operation
krb5kdc: starting...
W20260202 15:04:30.509922 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.993s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:30 dist-test-slave-6zzp krb5kdc[11493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:30.655776 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34673
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:30.767459 11509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:30.767694 11509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:30.767750 11509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:30.771674 11509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:30.771746 11509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:30.771775 11509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:30.771837 11509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:30.771881 11509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:30.776175 11509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:30.777339 11509 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:30.778240 11509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:30.783963 11515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:30.783953 11514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:30.783953 11517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:30.784296 11509 server_base.cc:1060] running on GCE node
I20260202 15:04:30.784824 11509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:30.785802 11509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:30.786998 11509 hybrid_clock.cc:648] HybridClock initialized: now 1770044670786977 us; error 48 us; skew 500 ppm
W20260202 15:04:30.790160 11509 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-02-02T15:04:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2299 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:30 dist-test-slave-6zzp krb5kdc[11522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:30 dist-test-slave-6zzp krb5kdc[11522](info): set up 2 sockets
Feb 02 15:04:30 dist-test-slave-6zzp krb5kdc[11522](info): commencing operation
krb5kdc: starting...
W20260202 15:04:32.818077 2705 mini_kdc.cc:121] Time spent starting KDC: real 2.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:32 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:32.963902 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42593
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:42593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:33.071381 11538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:33.071679 11538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:33.071776 11538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:33.075132 11538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:33.075246 11538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:33.075295 11538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:33.075362 11538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:33.075424 11538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:33.079842 11538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:42593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:33.081084 11538 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:33.082103 11538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:33.088455 11543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:33.088445 11544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:33.088672 11538 server_base.cc:1060] running on GCE node
W20260202 15:04:33.088439 11546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:33.089340 11538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:33.090237 11538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:33.091398 11538 hybrid_clock.cc:648] HybridClock initialized: now 1770044673091355 us; error 57 us; skew 500 ppm
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:33.094144 11538 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:04:33.095250 11538 webserver.cc:492] Webserver started at http://127.2.164.126:40099/ using document root <none> and password file <none>
I20260202 15:04:33.095834 11538 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:33.095885 11538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:33.096112 11538 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:33.097910 11538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "ba2e49dba13a484ab582915b503354da"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "da15f58a00502af059a084a886d397da"
server_key_iv: "bc67f5872a2507abc33069d98152a60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.098439 11538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "ba2e49dba13a484ab582915b503354da"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "da15f58a00502af059a084a886d397da"
server_key_iv: "bc67f5872a2507abc33069d98152a60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.101939 11538 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:33.104658 11553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.105870 11538 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:04:33.106196 11538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "ba2e49dba13a484ab582915b503354da"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "da15f58a00502af059a084a886d397da"
server_key_iv: "bc67f5872a2507abc33069d98152a60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.106353 11538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:33.132095 11538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:33.139876 11538 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:33.140118 11538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:33.148026 11538 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:42593
I20260202 15:04:33.148026 11605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:42593 every 8 connection(s)
I20260202 15:04:33.149143 11538 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:33.150520 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 11538
I20260202 15:04:33.150666 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:33.150905 2705 external_mini_cluster.cc:1468] Setting key f03fdfa02a7a00da738aae82acf9bdf0
I20260202 15:04:33.152100 11606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:33.158299 11606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Bootstrap starting.
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:33.160692 11606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:33.161648 11606 log.cc:826] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:33.163504 11606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: No bootstrap required, opened a new log
I20260202 15:04:33.164773 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.152174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.2.164.126:42593)
0202 15:04:33.152606 (+ 432us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:33.152617 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:33.152656 (+ 39us) server_negotiation.cc:408] Connection header received
0202 15:04:33.154079 (+ 1423us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:33.154104 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:33.154484 (+ 380us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:33.154839 (+ 355us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:33.155636 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.156774 (+ 1138us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.157560 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.157816 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.159957 (+ 2141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:33.160001 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:33.160013 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:33.160055 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:33.162259 (+ 2204us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.162729 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.162741 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.162746 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.162816 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.163072 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.163076 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.163078 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.163429 (+ 351us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:33.163647 (+ 218us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:33.163926 (+ 279us) server_negotiation.cc:300] Negotiation successful
0202 15:04:33.164182 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260202 15:04:33.167106 11606 raft_consensus.cc:359] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:33.167313 11606 raft_consensus.cc:385] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:33.167397 11606 raft_consensus.cc:740] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2e49dba13a484ab582915b503354da, State: Initialized, Role: FOLLOWER
I20260202 15:04:33.167827 11606 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:33.167973 11606 raft_consensus.cc:399] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:33.168051 11606 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:33.168169 11606 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:33.169489 11606 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:33.169823 11606 leader_election.cc:304] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba2e49dba13a484ab582915b503354da; no voters:
I20260202 15:04:33.170126 11606 leader_election.cc:290] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:33.170297 11611 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:33.170594 11611 raft_consensus.cc:697] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 LEADER]: Becoming Leader. State: Replica: ba2e49dba13a484ab582915b503354da, State: Running, Role: LEADER
I20260202 15:04:33.170960 11611 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:33.171244 11606 sys_catalog.cc:565] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:33.172720 11613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba2e49dba13a484ab582915b503354da. Latest consensus state: current_term: 1 leader_uuid: "ba2e49dba13a484ab582915b503354da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } } }
I20260202 15:04:33.172731 11612 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba2e49dba13a484ab582915b503354da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } } }
I20260202 15:04:33.172863 11613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:33.172865 11612 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:33.173300 11621 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:33.176723 11621 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:33.182062 11621 catalog_manager.cc:1357] Generated new cluster ID: 2ce01b7789be49ac95fdc9d8a737aa7c
I20260202 15:04:33.182135 11621 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:33.189832 11621 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:33.191006 11621 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:33.205125 11621 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Generated new TSK 0
I20260202 15:04:33.205996 11621 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:33.255802 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42593
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:33.364012 11634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:33.364257 11634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:33.364317 11634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:33.367561 11634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:33.367633 11634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:33.367712 11634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:33.372218 11634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:33.373577 11634 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:33.374433 11634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:33.381174 11640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:33.381204 11642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:33.381156 11639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:33.381603 11634 server_base.cc:1060] running on GCE node
I20260202 15:04:33.381953 11634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:33.382520 11634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:33.383715 11634 hybrid_clock.cc:648] HybridClock initialized: now 1770044673383672 us; error 66 us; skew 500 ppm
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:33.386483 11634 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:04:33.387553 11634 webserver.cc:492] Webserver started at http://127.2.164.65:44879/ using document root <none> and password file <none>
I20260202 15:04:33.388079 11634 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:33.388127 11634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:33.388288 11634 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:33.390044 11634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "e812a778a88249e399e4d2b7aa6c99c9"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "ce64464bb629bef01001b95980a2a3f4"
server_key_iv: "044bf0968a7dd0424e3dd979176df683"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.390493 11634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "e812a778a88249e399e4d2b7aa6c99c9"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "ce64464bb629bef01001b95980a2a3f4"
server_key_iv: "044bf0968a7dd0424e3dd979176df683"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.393886 11634 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:04:33.395977 11649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.396981 11634 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:33.397137 11634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "e812a778a88249e399e4d2b7aa6c99c9"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "ce64464bb629bef01001b95980a2a3f4"
server_key_iv: "044bf0968a7dd0424e3dd979176df683"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.397215 11634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:33.406213 11634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:33.409044 11634 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:33.409205 11634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:33.409688 11634 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:33.410609 11634 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:33.410656 11634 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.410696 11634 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:33.410717 11634 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.420178 11634 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:46701
I20260202 15:04:33.420200 11762 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:46701 every 8 connection(s)
I20260202 15:04:33.421193 11634 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:33.421911 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 11634
I20260202 15:04:33.422051 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:33.422326 2705 external_mini_cluster.cc:1468] Setting key e44e6c619c0394da3a2b9373aa8889de
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:04:33.434687 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.422981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:33501 (local address 127.2.164.126:42593)
0202 15:04:33.423130 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:33.423135 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:33.423889 (+ 754us) server_negotiation.cc:408] Connection header received
0202 15:04:33.424757 (+ 868us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:33.424763 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:33.424827 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:33.424921 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:33.426877 (+ 1956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.427466 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.428445 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.428582 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.431117 (+ 2535us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:33.431146 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:33.431148 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:33.431173 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:33.432576 (+ 1403us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.433168 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.433172 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.433174 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.433224 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.433523 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.433526 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.433528 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.433676 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:33.433791 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:33.434347 (+ 556us) server_negotiation.cc:300] Negotiation successful
0202 15:04:33.434507 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260202 15:04:33.435398 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.423294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:33501)
0202 15:04:33.423740 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:33.423781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:33.424556 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:33.425163 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:33.425171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:33.425560 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:33.426721 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:33.426742 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.427602 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.427608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:33.428307 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:33.428322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.428568 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.429266 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:33.429321 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:33.430962 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:33.432719 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:33.432724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:33.432736 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:33.433072 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:33.433317 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:33.433321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:33.433323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:33.433423 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:33.433833 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:33.433840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:33.434066 (+ 226us) client_negotiation.cc:769] Sending connection context
0202 15:04:33.434244 (+ 178us) client_negotiation.cc:240] Negotiation successful
0202 15:04:33.434584 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":127,"threads_started":1}
I20260202 15:04:33.436690 11763 heartbeater.cc:344] Connected to a master server at 127.2.164.126:42593
I20260202 15:04:33.436947 11763 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:33.437721 11763 heartbeater.cc:507] Master 127.2.164.126:42593 requested a full tablet report, sending...
I20260202 15:04:33.439582 11570 ts_manager.cc:194] Registered new tserver with Master: e812a778a88249e399e4d2b7aa6c99c9 (127.2.164.65:46701)
I20260202 15:04:33.441398 11570 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:33501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:33.476461 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42593
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:33.589802 11770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:33.590015 11770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:33.590072 11770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:33.593446 11770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:33.593508 11770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:33.593644 11770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:33.597817 11770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:33.598941 11770 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:33.599799 11770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:33.607724 11775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:33.607790 11770 server_base.cc:1060] running on GCE node
W20260202 15:04:33.607739 11776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:33.607749 11778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:33.608477 11770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:33.609280 11770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:33.610486 11770 hybrid_clock.cc:648] HybridClock initialized: now 1770044673610460 us; error 31 us; skew 500 ppm
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:33.613320 11770 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:04:33.614415 11770 webserver.cc:492] Webserver started at http://127.2.164.66:40183/ using document root <none> and password file <none>
I20260202 15:04:33.614998 11770 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:33.615073 11770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:33.615289 11770 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:33.617163 11770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "7e910b438fd54054ad1d6e040b90bae0"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "a77cc893d4db822bbbf2ecdabd7b35f9"
server_key_iv: "89557ea9711772239ba15ee11772f805"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.617681 11770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "7e910b438fd54054ad1d6e040b90bae0"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "a77cc893d4db822bbbf2ecdabd7b35f9"
server_key_iv: "89557ea9711772239ba15ee11772f805"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.621433 11770 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260202 15:04:33.623662 11785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.624701 11770 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:33.624843 11770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "7e910b438fd54054ad1d6e040b90bae0"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "a77cc893d4db822bbbf2ecdabd7b35f9"
server_key_iv: "89557ea9711772239ba15ee11772f805"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.624953 11770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:33.636155 11770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:33.639329 11770 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:33.639565 11770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:33.640168 11770 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:33.641207 11770 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:33.641278 11770 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.641352 11770 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:33.641397 11770 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.651032 11770 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:38341
I20260202 15:04:33.651064 11898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:38341 every 8 connection(s)
I20260202 15:04:33.651995 11770 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:33.652400 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 11770
I20260202 15:04:33.652556 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:33.652837 2705 external_mini_cluster.cc:1468] Setting key 8d56e2b9fef1a80191d8c6f097511fd3
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:04:33.665370 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.654249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:41079 (local address 127.2.164.126:42593)
0202 15:04:33.654417 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:33.654422 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:33.654862 (+ 440us) server_negotiation.cc:408] Connection header received
0202 15:04:33.655741 (+ 879us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:33.655745 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:33.655808 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:33.655893 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:33.657601 (+ 1708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.658128 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.658781 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.658962 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.661453 (+ 2491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:33.661477 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:33.661481 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:33.661521 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:33.663173 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.663740 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.663742 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.663744 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.663790 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.664111 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.664114 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.664116 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.664335 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:33.664442 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:33.665049 (+ 607us) server_negotiation.cc:300] Negotiation successful
0202 15:04:33.665183 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260202 15:04:33.666006 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.654219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:41079)
0202 15:04:33.654728 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:33.654763 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:33.655542 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:33.656043 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:33.656051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:33.656444 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:33.657454 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:33.657469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.658251 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.658257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:33.658662 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:33.658669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.658910 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.659579 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:33.659604 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:33.661290 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:33.663313 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:33.663319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:33.663332 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:33.663587 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:33.663886 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:33.663890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:33.663892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:33.663995 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:33.664441 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:33.664450 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:33.664719 (+ 269us) client_negotiation.cc:769] Sending connection context
0202 15:04:33.664962 (+ 243us) client_negotiation.cc:240] Negotiation successful
0202 15:04:33.665292 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
I20260202 15:04:33.667299 11899 heartbeater.cc:344] Connected to a master server at 127.2.164.126:42593
I20260202 15:04:33.667569 11899 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:33.668243 11899 heartbeater.cc:507] Master 127.2.164.126:42593 requested a full tablet report, sending...
I20260202 15:04:33.669637 11570 ts_manager.cc:194] Registered new tserver with Master: 7e910b438fd54054ad1d6e040b90bae0 (127.2.164.66:38341)
I20260202 15:04:33.670516 11570 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:41079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:33.704564 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42593
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:33.815227 11906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:33.815447 11906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:33.815546 11906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:33.818913 11906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:33.818984 11906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:33.819099 11906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:33.823451 11906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:33.824599 11906 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:33.825520 11906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:33.832365 11911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:33.832362 11914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:33.832365 11912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:33.832692 11906 server_base.cc:1060] running on GCE node
I20260202 15:04:33.833109 11906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:33.833683 11906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:33.834872 11906 hybrid_clock.cc:648] HybridClock initialized: now 1770044673834845 us; error 50 us; skew 500 ppm
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:33.837800 11906 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:04:33.838922 11906 webserver.cc:492] Webserver started at http://127.2.164.67:33451/ using document root <none> and password file <none>
I20260202 15:04:33.839507 11906 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:33.839586 11906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:33.839802 11906 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:33.841817 11906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "09e277711c9a4db49afa0579aff12c74"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "319eb4f18e8f0de7ab2774099f3547d5"
server_key_iv: "399a9488ff68d9fd05c2a05c7ced5956"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.842394 11906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "09e277711c9a4db49afa0579aff12c74"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "319eb4f18e8f0de7ab2774099f3547d5"
server_key_iv: "399a9488ff68d9fd05c2a05c7ced5956"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.846313 11906 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260202 15:04:33.848681 11921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.849834 11906 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:33.849980 11906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "09e277711c9a4db49afa0579aff12c74"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "319eb4f18e8f0de7ab2774099f3547d5"
server_key_iv: "399a9488ff68d9fd05c2a05c7ced5956"
server_key_version: "encryptionkey@0"
I20260202 15:04:33.850111 11906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:33.886785 11906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:33.889647 11906 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:33.889809 11906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:33.890312 11906 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:33.891261 11906 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:33.891311 11906 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.891352 11906 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:33.891368 11906 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:33.900517 11906 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:40597
I20260202 15:04:33.900554 12034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:40597 every 8 connection(s)
I20260202 15:04:33.901638 11906 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
Feb 02 15:04:33 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044673, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:33.911278 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 11906
I20260202 15:04:33.911412 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:33.911707 2705 external_mini_cluster.cc:1468] Setting key 1bb49edba4a527cd810d5e23b51f6dff
I20260202 15:04:33.914115 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.903365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:38029 (local address 127.2.164.126:42593)
0202 15:04:33.903496 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:33.903499 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:33.904132 (+ 633us) server_negotiation.cc:408] Connection header received
0202 15:04:33.905018 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:33.905023 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:33.905087 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:33.905199 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:33.906809 (+ 1610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.907313 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.907969 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.908147 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.910317 (+ 2170us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:33.910337 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:33.910339 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:33.910364 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:33.911967 (+ 1603us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.912508 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.912514 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.912516 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.912569 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:33.912881 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:33.912887 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:33.912890 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:33.913224 (+ 334us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:33.913325 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:33.913863 (+ 538us) server_negotiation.cc:300] Negotiation successful
0202 15:04:33.913995 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260202 15:04:33.914839 12037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:33.903583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:38029)
0202 15:04:33.903992 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:33.904026 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:33.904796 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:33.905354 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:33.905362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:33.905734 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:33.906656 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:33.906670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.907444 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:33.907450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:33.907850 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:33.907857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:33.908083 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:33.908663 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:33.908686 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:33.910168 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:33.912094 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:33.912122 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:33.912134 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:33.912405 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:33.912669 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:33.912672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:33.912674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:33.912782 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:33.913337 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:33.913343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:33.913575 (+ 232us) client_negotiation.cc:769] Sending connection context
0202 15:04:33.913811 (+ 236us) client_negotiation.cc:240] Negotiation successful
0202 15:04:33.914089 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260202 15:04:33.915903 12035 heartbeater.cc:344] Connected to a master server at 127.2.164.126:42593
I20260202 15:04:33.916162 12035 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:33.916687 12035 heartbeater.cc:507] Master 127.2.164.126:42593 requested a full tablet report, sending...
I20260202 15:04:33.917891 11570 ts_manager.cc:194] Registered new tserver with Master: 09e277711c9a4db49afa0579aff12c74 (127.2.164.67:40597)
I20260202 15:04:33.918509 11570 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:38029
I20260202 15:04:33.925606 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
WARNING: no policy specified for oryx/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
WARNING: no policy specified for oryx/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
I20260202 15:04:33.993886 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 11634
I20260202 15:04:33.999724 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 11770
I20260202 15:04:34.005234 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 11906
I20260202 15:04:34.010905 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 11538
I20260202 15:04:34.017580 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42593
--webserver_interface=127.2.164.126
--webserver_port=40099
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:42593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:34.126284 12044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:34.126518 12044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:34.126631 12044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:34.129933 12044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:34.129992 12044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:34.130015 12044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:34.130091 12044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:34.130123 12044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:34.134348 12044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:42593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=40099
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:34.135473 12044 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:34.136360 12044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:34.142153 12050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.142135 12049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.142122 12052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:34.142434 12044 server_base.cc:1060] running on GCE node
I20260202 15:04:34.142971 12044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:34.143858 12044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:34.145092 12044 hybrid_clock.cc:648] HybridClock initialized: now 1770044674145060 us; error 53 us; skew 500 ppm
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:34.147593 12044 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:04:34.148655 12044 webserver.cc:492] Webserver started at http://127.2.164.126:40099/ using document root <none> and password file <none>
I20260202 15:04:34.149281 12044 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:34.149379 12044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:34.153069 12044 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260202 15:04:34.155143 12059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.156311 12044 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:34.156457 12044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "ba2e49dba13a484ab582915b503354da"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "da15f58a00502af059a084a886d397da"
server_key_iv: "bc67f5872a2507abc33069d98152a60a"
server_key_version: "encryptionkey@0"
I20260202 15:04:34.156903 12044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:34.171597 12044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:34.174355 12044 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:34.174551 12044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:34.181941 12044 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:42593
I20260202 15:04:34.181949 12111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:42593 every 8 connection(s)
I20260202 15:04:34.182989 12044 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:34.183269 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12044
I20260202 15:04:34.183758 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:46701
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:42593
--webserver_port=44879
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.164.65
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260202 15:04:34.188355 12112 sys_catalog.cc:263] Verifying existing consensus state
I20260202 15:04:34.190747 12112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Bootstrap starting.
I20260202 15:04:34.211784 12112 log.cc:826] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:34.219611 12112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260202 15:04:34.220023 12112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Bootstrap complete.
I20260202 15:04:34.224529 12112 raft_consensus.cc:359] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:34.224918 12112 raft_consensus.cc:740] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2e49dba13a484ab582915b503354da, State: Initialized, Role: FOLLOWER
I20260202 15:04:34.225481 12112 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:34.225620 12112 raft_consensus.cc:399] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:34.225683 12112 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:34.225771 12112 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 1 FOLLOWER]: Advancing to term 2
I20260202 15:04:34.227737 12112 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:34.228108 12112 leader_election.cc:304] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba2e49dba13a484ab582915b503354da; no voters:
I20260202 15:04:34.228448 12112 leader_election.cc:290] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [CANDIDATE]: Term 2 election: Requested vote from peers
I20260202 15:04:34.228688 12116 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 2 FOLLOWER]: Leader election won for term 2
I20260202 15:04:34.228935 12116 raft_consensus.cc:697] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [term 2 LEADER]: Becoming Leader. State: Replica: ba2e49dba13a484ab582915b503354da, State: Running, Role: LEADER
I20260202 15:04:34.229276 12116 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } }
I20260202 15:04:34.229609 12112 sys_catalog.cc:565] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:34.230556 12118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba2e49dba13a484ab582915b503354da. Latest consensus state: current_term: 2 leader_uuid: "ba2e49dba13a484ab582915b503354da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } } }
I20260202 15:04:34.230671 12118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:34.230960 12117 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba2e49dba13a484ab582915b503354da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2e49dba13a484ab582915b503354da" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42593 } } }
I20260202 15:04:34.231034 12117 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:34.231377 12125 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:34.233958 12125 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:34.234508 12125 catalog_manager.cc:1269] Loaded cluster ID: 2ce01b7789be49ac95fdc9d8a737aa7c
I20260202 15:04:34.234580 12125 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:34.236394 12125 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:34.236979 12125 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ba2e49dba13a484ab582915b503354da: Loaded TSK: 0
I20260202 15:04:34.237999 12125 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260202 15:04:34.319976 12114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:34.320206 12114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:34.320304 12114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:34.323666 12114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:34.323741 12114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:34.323864 12114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:34.328147 12114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:46701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=44879
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:34.329371 12114 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:34.330503 12114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:34.337193 12139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.337189 12138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.337822 12141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:34.338012 12114 server_base.cc:1060] running on GCE node
I20260202 15:04:34.338420 12114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:34.339046 12114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:34.340216 12114 hybrid_clock.cc:648] HybridClock initialized: now 1770044674340187 us; error 42 us; skew 500 ppm
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044674, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:34.343305 12114 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260202 15:04:34.344660 12114 webserver.cc:492] Webserver started at http://127.2.164.65:44879/ using document root <none> and password file <none>
I20260202 15:04:34.345362 12114 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:34.345454 12114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:34.349846 12114 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:04:34.352057 12148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.353303 12114 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260202 15:04:34.353420 12114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "e812a778a88249e399e4d2b7aa6c99c9"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "ce64464bb629bef01001b95980a2a3f4"
server_key_iv: "044bf0968a7dd0424e3dd979176df683"
server_key_version: "encryptionkey@0"
I20260202 15:04:34.353925 12114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:34.393682 12114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:34.397248 12114 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:34.397468 12114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:34.398200 12114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:34.399317 12114 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:34.399384 12114 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.399451 12114 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:34.399487 12114 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.411715 12114 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:46701
I20260202 15:04:34.411772 12261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:46701 every 8 connection(s)
I20260202 15:04:34.413128 12114 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:34.421278 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12114
I20260202 15:04:34.421767 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:38341
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:42593
--webserver_port=40183
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.164.66
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.425992 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.415246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:60453)
0202 15:04:34.415776 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.415823 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.416812 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.418017 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.418027 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.418573 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.419817 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.419836 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.420954 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.420958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.421606 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.421616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.421822 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.422459 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.422490 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.425192 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":130,"threads_started":1}
W20260202 15:04:34.425997 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.415236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:60453 (local address 127.2.164.126:42593)
0202 15:04:34.415769 (+ 533us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.415780 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.415980 (+ 200us) server_negotiation.cc:408] Connection header received
0202 15:04:34.417258 (+ 1278us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.417269 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.417622 (+ 353us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.417860 (+ 238us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.419991 (+ 2131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.420815 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.421740 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.422268 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.425380 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:60453: BlockingRecv error: recv got EOF from 127.2.164.65:60453 (error 108)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":131,"threads_started":1}
W20260202 15:04:34.426858 12262 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:42593 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.164.126:42593: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.433219 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.427340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:56661)
0202 15:04:34.427609 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.427627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.427734 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.428010 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.428016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.428218 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.428473 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.428484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.429516 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.429522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.430172 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.430183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.430315 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.430939 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.430961 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.433099 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170}
W20260202 15:04:34.433498 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.427450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:56661 (local address 127.2.164.126:42593)
0202 15:04:34.427568 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.427572 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.427697 (+ 125us) server_negotiation.cc:408] Connection header received
0202 15:04:34.427835 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.427838 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.427884 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.427969 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.428610 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.429378 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.430331 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.430517 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.433229 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:56661: BlockingRecv error: recv got EOF from 127.2.164.65:56661 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.439896 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.433728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:38805)
0202 15:04:34.434041 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.434057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.434146 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.434479 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.434483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.434706 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.434970 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.434977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.435981 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.435986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.436620 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.436630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.437104 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.437652 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.437671 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.439754 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":115,"threads_started":1}
W20260202 15:04:34.440040 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.434125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:38805 (local address 127.2.164.126:42593)
0202 15:04:34.434241 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.434245 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.434258 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:34.434291 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.434295 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.434345 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.434436 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.435109 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.435849 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.436762 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.436959 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.439926 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:38805: BlockingRecv error: recv got EOF from 127.2.164.65:38805 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260202 15:04:34.440444 12262 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260202 15:04:34.533095 12266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:34.533329 12266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:34.533434 12266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:34.536934 12266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:34.537060 12266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:34.537158 12266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:34.542263 12266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:38341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=40183
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:34.543655 12266 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:34.544754 12266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:34.552039 12272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.552059 12273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.552081 12275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:34.552194 12266 server_base.cc:1060] running on GCE node
I20260202 15:04:34.552731 12266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:34.553262 12266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:34.554412 12266 hybrid_clock.cc:648] HybridClock initialized: now 1770044674554397 us; error 27 us; skew 500 ppm
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044674, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:34.556869 12266 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260202 15:04:34.557886 12266 webserver.cc:492] Webserver started at http://127.2.164.66:40183/ using document root <none> and password file <none>
I20260202 15:04:34.558494 12266 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:34.558557 12266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:34.562126 12266 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260202 15:04:34.563995 12282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.565171 12266 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:34.565287 12266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "7e910b438fd54054ad1d6e040b90bae0"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "a77cc893d4db822bbbf2ecdabd7b35f9"
server_key_iv: "89557ea9711772239ba15ee11772f805"
server_key_version: "encryptionkey@0"
I20260202 15:04:34.565701 12266 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:34.581743 12266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:34.584887 12266 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:34.585160 12266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:34.585757 12266 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:34.586653 12266 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:34.586702 12266 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.586772 12266 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:34.586813 12266 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.596470 12266 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:38341
I20260202 15:04:34.596487 12395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:38341 every 8 connection(s)
I20260202 15:04:34.597527 12266 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:34.598953 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12266
I20260202 15:04:34.599396 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:40597
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:42593
--webserver_port=33451
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.164.67
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260202 15:04:34.609349 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.599544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:41625 (local address 127.2.164.126:42593)
0202 15:04:34.600672 (+ 1128us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.600680 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.600702 (+ 22us) server_negotiation.cc:408] Connection header received
0202 15:04:34.601572 (+ 870us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.601575 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.601627 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.601703 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.604052 (+ 2349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.604554 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.605595 (+ 1041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.605725 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.609218 (+ 3493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:41625: BlockingRecv error: recv got EOF from 127.2.164.66:41625 (error 108)
Metrics: {"server-negotiator.queue_time_us":994}
I20260202 15:04:34.609866 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.599856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:41625)
0202 15:04:34.600378 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.600429 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.601322 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.601896 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.601909 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.602484 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.603862 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.603886 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.604690 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.604696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.605471 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.605484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.605746 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.606394 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.606437 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.609073 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":103,"threads_started":1}
W20260202 15:04:34.610321 12396 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:42593 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.164.126:42593: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.615934 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.610824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:45557)
0202 15:04:34.611037 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.611053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.611171 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.611442 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.611446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.611628 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.611878 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.611886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.612604 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.612610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.613264 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.613275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.613392 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.613880 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.613898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.615813 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260202 15:04:34.616052 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.610911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45557 (local address 127.2.164.126:42593)
0202 15:04:34.611034 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.611038 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.611104 (+ 66us) server_negotiation.cc:408] Connection header received
0202 15:04:34.611282 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.611285 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.611330 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.611403 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.612007 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.612489 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.613374 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.613512 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.615936 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:45557: BlockingRecv error: recv got EOF from 127.2.164.66:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.621308 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.616581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:45169)
0202 15:04:34.616696 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.616712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.616787 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.617053 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.617055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.617196 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.617351 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.617356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.618297 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.618299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.618639 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.618645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.618730 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.619265 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.619281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.621181 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260202 15:04:34.621429 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.616661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45169 (local address 127.2.164.126:42593)
0202 15:04:34.616775 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.616779 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.616793 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:34.616867 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.616870 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.616916 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.616991 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.617462 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.618190 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.618750 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.618936 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.621302 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:45169: BlockingRecv error: recv got EOF from 127.2.164.66:45169 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260202 15:04:34.621592 12396 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260202 15:04:34.715869 12399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:34.716181 12399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:34.716281 12399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:34.719736 12399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:34.719834 12399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:34.719997 12399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:34.724344 12399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:40597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=33451
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:34.725581 12399 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:34.726468 12399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:34.733385 12407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.733436 12404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:34.733481 12405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:34.733989 12399 server_base.cc:1060] running on GCE node
I20260202 15:04:34.734432 12399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:34.735095 12399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:34.736291 12399 hybrid_clock.cc:648] HybridClock initialized: now 1770044674736262 us; error 44 us; skew 500 ppm
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044674, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:34.738865 12399 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260202 15:04:34.739933 12399 webserver.cc:492] Webserver started at http://127.2.164.67:33451/ using document root <none> and password file <none>
I20260202 15:04:34.740476 12399 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:34.740543 12399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:34.744081 12399 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:34.746104 12414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.747200 12399 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:34.747296 12399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "09e277711c9a4db49afa0579aff12c74"
format_stamp: "Formatted at 2026-02-02 15:04:33 on dist-test-slave-6zzp"
server_key: "319eb4f18e8f0de7ab2774099f3547d5"
server_key_iv: "399a9488ff68d9fd05c2a05c7ced5956"
server_key_version: "encryptionkey@0"
I20260202 15:04:34.747713 12399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:34.765806 12399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:34.768591 12399 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:34.768750 12399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:34.769338 12399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:34.770303 12399 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:34.770352 12399 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.770393 12399 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:34.770408 12399 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:34.780390 12399 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:40597
I20260202 15:04:34.780422 12527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:40597 every 8 connection(s)
I20260202 15:04:34.781546 12399 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:34.786099 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12399
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260202 15:04:34.792313 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.783406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59727 (local address 127.2.164.126:42593)
0202 15:04:34.783595 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.783599 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.784329 (+ 730us) server_negotiation.cc:408] Connection header received
0202 15:04:34.785156 (+ 827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.785159 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.785211 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.785291 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.787264 (+ 1973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.787763 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.788436 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.788611 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.792149 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:59727: BlockingRecv error: recv got EOF from 127.2.164.67:59727 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260202 15:04:34.792753 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.783614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:59727)
0202 15:04:34.784147 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.784193 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.784933 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.785453 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.785461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.785844 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.787099 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.787115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.787889 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.787894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.788313 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.788321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.788559 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.789282 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.789320 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.791983 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
W20260202 15:04:34.793249 12528 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:42593 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.164.126:42593: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
I20260202 15:04:34.795632 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.787085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.2.164.126:42593)
0202 15:04:34.787354 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.787361 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.787375 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:34.787423 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.787425 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.787464 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.787563 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.788386 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.789135 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.790072 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.790278 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.791341 (+ 1063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:34.791372 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:34.791385 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:34.791428 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:34.793617 (+ 2189us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:34.794079 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:34.794085 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:34.794090 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:34.794202 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:34.794457 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:34.794459 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:34.794461 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:34.794791 (+ 330us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:34.794988 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:34.795236 (+ 248us) server_negotiation.cc:300] Negotiation successful
0202 15:04:34.795462 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.798686 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.793734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:42251)
0202 15:04:34.793855 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.793873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.793950 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.794239 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.794260 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.794431 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.794631 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.794637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.795557 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.795561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.795998 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.796006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.796093 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.796566 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.796580 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.798571 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260202 15:04:34.798832 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.793785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:42251 (local address 127.2.164.126:42593)
0202 15:04:34.793900 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.793903 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.793916 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:34.794062 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.794066 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.794112 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.794201 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.794738 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.795441 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.796103 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.796269 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.798688 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:42251: BlockingRecv error: recv got EOF from 127.2.164.67:42251 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:34 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:34.803311 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:34.799262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:42625)
0202 15:04:34.799382 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:34.799400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:34.799480 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:34.799677 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:34.799679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:34.799802 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:34.799957 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.799961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.800605 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.800608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:34.800954 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:34.800959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.801161 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.801637 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:34.801650 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:34.803208 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260202 15:04:34.803509 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:34.799340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:42625 (local address 127.2.164.126:42593)
0202 15:04:34.799464 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:34.799468 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:34.799480 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:34.799548 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:34.799550 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:34.799587 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:34.799649 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:34.800049 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.800516 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:34.801154 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:34.801283 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:34.803366 (+ 2083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:42625: BlockingRecv error: recv got EOF from 127.2.164.67:42625 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260202 15:04:34.803591 12528 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Feb 02 15:04:35 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:35 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:35.446308 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:35.441230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:50543)
0202 15:04:35.441535 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:35.441553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:35.441666 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:35.441924 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:35.441927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:35.442134 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:35.442365 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:35.442372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.443167 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:35.443171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:35.443564 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:35.443572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.443674 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:35.444269 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:35.444283 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:35.446125 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
W20260202 15:04:35.446411 12534 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:35.441406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50543 (local address 127.2.164.126:42593)
0202 15:04:35.441654 (+ 248us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:35.441657 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:35.441671 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:35.441740 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:35.441742 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:35.441787 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:35.441866 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:35.442476 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.442978 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:35.443666 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.443803 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:35.446276 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:50543: BlockingRecv error: recv got EOF from 127.2.164.65:50543 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
Feb 02 15:04:35 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:35 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:35.627781 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:35.622303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:41751)
0202 15:04:35.622626 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:35.622640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:35.622740 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:35.622974 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:35.622976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:35.623201 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:35.623455 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:35.623462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.624397 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:35.624401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:35.624833 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:35.624845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.624964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:35.625719 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:35.625734 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:35.627596 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
W20260202 15:04:35.627872 12534 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:35.622506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:41751 (local address 127.2.164.126:42593)
0202 15:04:35.622760 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:35.622764 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:35.622778 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:35.622813 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:35.622816 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:35.622860 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:35.622934 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:35.623587 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.624233 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:35.624979 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.625221 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:35.627749 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:41751: BlockingRecv error: recv got EOF from 127.2.164.66:41751 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Feb 02 15:04:35 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:35 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:35.809417 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:35.804260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:50827)
0202 15:04:35.804559 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:35.804574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:35.804666 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:35.804950 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:35.804953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:35.805178 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:35.805382 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:35.805389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.806109 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:35.806112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:35.806516 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:35.806524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.806628 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:35.807192 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:35.807205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:35.809278 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
W20260202 15:04:35.809571 12534 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:35.804458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:50827 (local address 127.2.164.126:42593)
0202 15:04:35.804598 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:35.804601 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:35.804616 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:04:35.804754 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:35.804757 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:35.804806 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:35.804880 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:35.805508 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.805990 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:35.806655 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:35.806846 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:35.809437 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:50827: BlockingRecv error: recv got EOF from 127.2.164.67:50827 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Feb 02 15:04:36 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:36 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:36.452521 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:36.447239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:44125)
0202 15:04:36.447560 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:36.447575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:36.447676 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:36.447944 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:36.447946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:36.448139 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:36.448335 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:36.448359 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.449381 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:36.449385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:36.449866 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:36.449875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.449967 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:36.450512 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:36.450525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:36.452388 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
W20260202 15:04:36.452667 12538 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:36.447390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:44125 (local address 127.2.164.126:42593)
0202 15:04:36.447665 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:36.447668 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:36.447683 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:04:36.447777 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:36.447780 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:36.447826 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:36.447905 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:36.448481 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.449210 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:36.449968 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.450101 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:36.452537 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:44125: BlockingRecv error: recv got EOF from 127.2.164.65:44125 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
Feb 02 15:04:36 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:36 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:36.633773 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:36.628659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:35435)
0202 15:04:36.628941 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:36.628963 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:36.629092 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:36.629342 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:36.629344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:36.629525 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:36.629719 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:36.629724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.630627 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:36.630630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:36.631002 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:36.631009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.631113 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:36.631690 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:36.631713 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:36.633603 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
W20260202 15:04:36.633818 12538 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:36.628754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35435 (local address 127.2.164.126:42593)
0202 15:04:36.628945 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:36.628948 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:36.629036 (+ 88us) server_negotiation.cc:408] Connection header received
0202 15:04:36.629171 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:36.629173 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:36.629237 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:36.629304 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:36.629855 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.630514 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:36.631115 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.631265 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:36.633719 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:35435: BlockingRecv error: recv got EOF from 127.2.164.66:35435 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Feb 02 15:04:36 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:36 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:36.815155 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:36.810127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:48779)
0202 15:04:36.810490 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:36.810505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:36.810607 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:36.810851 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:36.810853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:36.811044 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:36.811254 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:36.811261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.811989 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:36.811993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:36.812394 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:36.812402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.812535 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:36.813216 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:36.813230 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:36.815008 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":142,"threads_started":1}
W20260202 15:04:36.815339 12538 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:36.810293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:48779 (local address 127.2.164.126:42593)
0202 15:04:36.810436 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:36.810440 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:36.810572 (+ 132us) server_negotiation.cc:408] Connection header received
0202 15:04:36.810690 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:36.810692 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:36.810740 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:36.810812 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:36.811362 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.811858 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:36.812602 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:36.812766 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:36.815192 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:48779: BlockingRecv error: recv got EOF from 127.2.164.67:48779 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Feb 02 15:04:37 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:37 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:37.458138 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:37.453320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:50227)
0202 15:04:37.453653 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:37.453674 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:37.453773 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:37.454041 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:37.454044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:37.454236 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:37.454440 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:37.454447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.455158 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:37.455161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:37.455546 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:37.455553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.455642 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:37.456157 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:37.456170 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:37.458015 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":140,"threads_started":1}
W20260202 15:04:37.458323 12541 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:37.453347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50227 (local address 127.2.164.126:42593)
0202 15:04:37.453619 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:37.453622 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:37.453766 (+ 144us) server_negotiation.cc:408] Connection header received
0202 15:04:37.453839 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:37.453842 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:37.453890 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:37.453977 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:37.454550 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.455059 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:37.455641 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.455768 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:37.458219 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:50227: BlockingRecv error: recv got EOF from 127.2.164.65:50227 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
Feb 02 15:04:37 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:37 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:37.639846 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:37.634718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:48775)
0202 15:04:37.635028 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:37.635042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:37.635248 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:37.635520 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:37.635523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:37.635704 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:37.635897 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:37.635902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.636663 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:37.636667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:37.637170 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:37.637183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.637280 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:37.637851 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:37.637864 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:37.639725 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
W20260202 15:04:37.640058 12541 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:37.634778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:48775 (local address 127.2.164.126:42593)
0202 15:04:37.634932 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:37.634936 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:37.635182 (+ 246us) server_negotiation.cc:408] Connection header received
0202 15:04:37.635345 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:37.635348 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:37.635401 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:37.635472 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:37.636020 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.636558 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:37.637408 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.637626 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:37.639897 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:48775: BlockingRecv error: recv got EOF from 127.2.164.66:48775 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Feb 02 15:04:37 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:37 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:37.821051 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:37.815920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:44793)
0202 15:04:37.816280 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:37.816293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:37.816447 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:37.816718 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:37.816721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:37.816941 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:37.817157 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:37.817164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.817884 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:37.817889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:37.818279 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:37.818285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.818420 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:37.818948 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:37.818960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:37.820856 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":146,"threads_started":1}
W20260202 15:04:37.821144 12541 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:37.816036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:44793 (local address 127.2.164.126:42593)
0202 15:04:37.816212 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:37.816218 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:37.816371 (+ 153us) server_negotiation.cc:408] Connection header received
0202 15:04:37.816549 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:37.816554 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:37.816609 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:37.816697 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:37.817281 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.817774 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:37.818397 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:37.818588 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:37.820991 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:44793: BlockingRecv error: recv got EOF from 127.2.164.67:44793 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Feb 02 15:04:38 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:38 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:38.469257 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:38.459223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:38391)
0202 15:04:38.459515 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:38.459529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:38.459676 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:38.459963 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:38.459966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:38.460147 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:38.460383 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:38.460390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.461287 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:38.461290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:38.461731 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:38.461740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.461832 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:38.466957 (+ 5125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:38.466975 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:38.469122 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260202 15:04:38.469492 12546 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:38.459293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:38391 (local address 127.2.164.126:42593)
0202 15:04:38.459611 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:38.459616 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:38.459642 (+ 26us) server_negotiation.cc:408] Connection header received
0202 15:04:38.459777 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:38.459780 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:38.459832 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:38.459925 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:38.460517 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.461157 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:38.461838 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.461977 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:38.469326 (+ 7349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:38391: BlockingRecv error: recv got EOF from 127.2.164.65:38391 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":117,"threads_started":1}
Feb 02 15:04:38 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:38 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:38.646081 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:38.640877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:60789)
0202 15:04:38.641182 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:38.641201 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:38.641372 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:38.641661 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:38.641664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:38.641874 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:38.642068 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:38.642074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.642815 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:38.642821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:38.643203 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:38.643210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.643342 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:38.643901 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:38.643915 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:38.645959 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
W20260202 15:04:38.646266 12546 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:38.640968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60789 (local address 127.2.164.126:42593)
0202 15:04:38.641174 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:38.641178 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:38.641284 (+ 106us) server_negotiation.cc:408] Connection header received
0202 15:04:38.641490 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:38.641493 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:38.641549 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:38.641637 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:38.642198 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.642704 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:38.643333 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.643551 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:38.646103 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:60789: BlockingRecv error: recv got EOF from 127.2.164.66:60789 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Feb 02 15:04:38 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:38 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:38.827250 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:38.821906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:52031)
0202 15:04:38.822324 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:38.822338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:38.822433 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:38.822679 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:38.822681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:38.822868 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:38.823070 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:38.823076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.824052 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:38.824056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:38.824490 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:38.824499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.824616 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:38.825288 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:38.825303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:38.827105 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":98,"threads_started":1}
W20260202 15:04:38.827443 12546 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:38.822003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:52031 (local address 127.2.164.126:42593)
0202 15:04:38.822202 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:38.822214 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:38.822414 (+ 200us) server_negotiation.cc:408] Connection header received
0202 15:04:38.822520 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:38.822523 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:38.822575 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:38.822653 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:38.823417 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.823921 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:38.824613 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:38.824744 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:38.827274 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:52031: BlockingRecv error: recv got EOF from 127.2.164.67:52031 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Feb 02 15:04:39 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:39 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:39.475009 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:39.470075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:43905)
0202 15:04:39.470352 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:39.470365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:39.470453 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:39.470796 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:39.470799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:39.470994 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:39.471195 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:39.471201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.471940 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:39.471942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:39.472338 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:39.472345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.472434 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:39.472970 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:39.472983 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:39.474890 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
W20260202 15:04:39.475171 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:39.470137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:43905 (local address 127.2.164.126:42593)
0202 15:04:39.470424 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:39.470427 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:39.470442 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:04:39.470624 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:39.470627 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:39.470675 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:39.470757 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:39.471329 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.471845 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:39.472439 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.472577 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:39.475034 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:43905: BlockingRecv error: recv got EOF from 127.2.164.65:43905 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":133,"threads_started":1}
Feb 02 15:04:39 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:39 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:39.652195 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:39.646903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:33907)
0202 15:04:39.647232 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:39.647246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:39.647368 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:39.647604 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:39.647606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:39.647791 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:39.647988 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:39.647994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.648785 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:39.648788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:39.649327 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:39.649338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.649459 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:39.650071 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:39.650087 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:39.652049 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":148,"threads_started":1}
W20260202 15:04:39.652328 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:39.647079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:33907 (local address 127.2.164.126:42593)
0202 15:04:39.647228 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:39.647232 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:39.647367 (+ 135us) server_negotiation.cc:408] Connection header received
0202 15:04:39.647444 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:39.647447 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:39.647498 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:39.647573 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:39.648126 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.648646 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:39.649451 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.649592 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:39.652182 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:33907: BlockingRecv error: recv got EOF from 127.2.164.66:33907 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Feb 02 15:04:39 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:39 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:39.833238 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:39.828233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:45811)
0202 15:04:39.828542 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:39.828556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:39.828661 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:39.828916 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:39.828919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:39.829127 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:39.829327 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:39.829334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.830037 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:39.830040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:39.830428 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:39.830434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.830538 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:39.831151 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:39.831165 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:39.833111 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
W20260202 15:04:39.833427 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:39.828334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45811 (local address 127.2.164.126:42593)
0202 15:04:39.828542 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:39.828546 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:39.828654 (+ 108us) server_negotiation.cc:408] Connection header received
0202 15:04:39.828731 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:39.828733 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:39.828778 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:39.828846 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:39.829452 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.829942 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:39.830535 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:39.830660 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:39.833261 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:45811: BlockingRecv error: recv got EOF from 127.2.164.67:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Feb 02 15:04:40 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:40 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:40.480788 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:40.475884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:58507)
0202 15:04:40.476170 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:40.476184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:40.476289 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:40.476556 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:40.476558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:40.476738 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:40.476938 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:40.476944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.477750 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:40.477753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:40.478149 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:40.478156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.478255 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:40.478817 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:40.478831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:40.480672 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260202 15:04:40.480919 12554 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:40.475963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:58507 (local address 127.2.164.126:42593)
0202 15:04:40.476255 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:40.476259 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:40.476276 (+ 17us) server_negotiation.cc:408] Connection header received
0202 15:04:40.476386 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:40.476389 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:40.476435 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:40.476526 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:40.477119 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.477649 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:40.478254 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.478382 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:40.480816 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:58507: BlockingRecv error: recv got EOF from 127.2.164.65:58507 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
Feb 02 15:04:40 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:40 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:40.658133 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:40.653150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:46903)
0202 15:04:40.653422 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:40.653435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:40.653543 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:40.653796 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:40.653799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:40.653975 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:40.654167 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:40.654172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.654887 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:40.654889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:40.655419 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:40.655428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.655566 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:40.656113 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:40.656131 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:40.657951 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
W20260202 15:04:40.658187 12554 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:40.653192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46903 (local address 127.2.164.126:42593)
0202 15:04:40.653344 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:40.653347 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:40.653510 (+ 163us) server_negotiation.cc:408] Connection header received
0202 15:04:40.653625 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:40.653628 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:40.653676 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:40.653754 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:40.654286 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.654786 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:40.655549 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.655781 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:40.658082 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:46903: BlockingRecv error: recv got EOF from 127.2.164.66:46903 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Feb 02 15:04:40 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:40 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:40.839639 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:40.834084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:36123)
0202 15:04:40.834345 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:40.834364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:40.834529 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:40.834895 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:40.834898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:40.835087 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:40.835294 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:40.835300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.836049 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:40.836054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:40.836446 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:40.836453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.836592 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:40.837513 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:40.837526 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:40.839512 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":120,"threads_started":1}
W20260202 15:04:40.839846 12554 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:40.834140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:36123 (local address 127.2.164.126:42593)
0202 15:04:40.834319 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:40.834325 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:40.834451 (+ 126us) server_negotiation.cc:408] Connection header received
0202 15:04:40.834630 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:40.834635 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:40.834695 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:40.834765 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:40.835416 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.835931 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:40.836572 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:40.836771 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:40.839704 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:36123: BlockingRecv error: recv got EOF from 127.2.164.67:36123 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Feb 02 15:04:41 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:41 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:41.487311 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:41.481923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:41701)
0202 15:04:41.482327 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:41.482350 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:41.482463 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:41.482726 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:41.482729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:41.482914 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:41.483111 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:41.483117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.483948 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:41.483952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:41.484376 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:41.484385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.484498 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:41.485153 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:41.485169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:41.487130 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":99,"threads_started":1}
W20260202 15:04:41.487525 12558 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:41.482041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41701 (local address 127.2.164.126:42593)
0202 15:04:41.482351 (+ 310us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:41.482355 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:41.482427 (+ 72us) server_negotiation.cc:408] Connection header received
0202 15:04:41.482557 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:41.482559 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:41.482606 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:41.482686 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:41.483251 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.483789 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:41.484492 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.484636 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:41.487341 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:41701: BlockingRecv error: recv got EOF from 127.2.164.65:41701 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
Feb 02 15:04:41 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:41 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:41.664598 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:41.658922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:57185)
0202 15:04:41.659283 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:41.659298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:41.659412 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:41.659797 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:41.659800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:41.660019 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:41.660219 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:41.660255 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.661091 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:41.661095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:41.661533 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:41.661541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.661639 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:41.662185 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:41.662199 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:41.664361 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
W20260202 15:04:41.664664 12558 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:41.659007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:57185 (local address 127.2.164.126:42593)
0202 15:04:41.659250 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:41.659263 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:41.659398 (+ 135us) server_negotiation.cc:408] Connection header received
0202 15:04:41.659621 (+ 223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:41.659624 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:41.659675 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:41.659746 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:41.660372 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.660880 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:41.661639 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.661792 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:41.664526 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:57185: BlockingRecv error: recv got EOF from 127.2.164.66:57185 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Feb 02 15:04:41 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:41 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:41.845757 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:41.840430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:40345)
0202 15:04:41.840803 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:41.840825 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:41.840955 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:41.841279 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:41.841282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:41.841616 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:41.841821 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:41.841829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.842644 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:41.842649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:41.843023 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:41.843030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.843171 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:41.843717 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:41.843731 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:41.845636 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":91,"threads_started":1}
W20260202 15:04:41.845963 12558 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:41.840564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40345 (local address 127.2.164.126:42593)
0202 15:04:41.840751 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:41.840756 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:41.840900 (+ 144us) server_negotiation.cc:408] Connection header received
0202 15:04:41.841116 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:41.841119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:41.841168 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:41.841257 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:41.841987 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.842519 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:41.843142 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:41.843346 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:41.845815 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:40345: BlockingRecv error: recv got EOF from 127.2.164.67:40345 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Feb 02 15:04:42 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:42 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:42.493484 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:42.488259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:47951)
0202 15:04:42.488558 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:42.488579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:42.488716 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:42.489088 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:42.489093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:42.489315 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:42.489580 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:42.489591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.490474 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:42.490478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:42.490870 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:42.490877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.490972 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:42.491558 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:42.491571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:42.493366 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
W20260202 15:04:42.493706 12562 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:42.488506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:47951 (local address 127.2.164.126:42593)
0202 15:04:42.488748 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:42.488752 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:42.488764 (+ 12us) server_negotiation.cc:408] Connection header received
0202 15:04:42.488829 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:42.488832 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:42.488883 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:42.488989 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:42.489729 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.490357 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:42.490987 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.491120 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:42.493547 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:47951: BlockingRecv error: recv got EOF from 127.2.164.65:47951 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
Feb 02 15:04:42 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:42 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:42.671697 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:42.665766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:51479)
0202 15:04:42.666048 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:42.666062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:42.666159 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:42.666433 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:42.666435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:42.666633 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:42.666826 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:42.666831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.668429 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:42.668435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:42.668842 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:42.668850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.668943 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:42.669507 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:42.669520 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:42.671516 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
W20260202 15:04:42.671736 12562 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:42.665905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:51479 (local address 127.2.164.126:42593)
0202 15:04:42.666084 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:42.666088 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:42.666110 (+ 22us) server_negotiation.cc:408] Connection header received
0202 15:04:42.666251 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:42.666254 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:42.666320 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:42.666393 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:42.667127 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.668198 (+ 1071us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:42.668943 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.669110 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:42.671636 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:51479: BlockingRecv error: recv got EOF from 127.2.164.66:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Feb 02 15:04:42 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:42 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:42.852058 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:42.846711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:42273)
0202 15:04:42.847178 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:42.847195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:42.847346 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:42.847625 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:42.847628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:42.847847 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:42.848037 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:42.848044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.848773 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:42.848780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:42.849232 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:42.849239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.849369 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:42.849939 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:42.849953 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:42.851881 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":94,"threads_started":1}
W20260202 15:04:42.852116 12562 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:42.846821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:42273 (local address 127.2.164.126:42593)
0202 15:04:42.846997 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:42.847001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:42.847271 (+ 270us) server_negotiation.cc:408] Connection header received
0202 15:04:42.847449 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:42.847455 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:42.847509 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:42.847606 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:42.848160 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.848663 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:42.849350 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:42.849556 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:42.852005 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:42273: BlockingRecv error: recv got EOF from 127.2.164.67:42273 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Feb 02 15:04:43 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:43 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:43.503638 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:43.494208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:41037)
0202 15:04:43.494559 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:43.494574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:43.494672 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:43.494928 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:43.494930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:43.495108 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:43.495331 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:43.495338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.496144 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:43.496148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:43.496762 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:43.496774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.496888 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:43.501431 (+ 4543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:43.501447 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:43.503503 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
W20260202 15:04:43.503863 12566 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:43.494327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41037 (local address 127.2.164.126:42593)
0202 15:04:43.494601 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:43.494605 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:43.494622 (+ 17us) server_negotiation.cc:408] Connection header received
0202 15:04:43.494763 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:43.494766 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:43.494813 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:43.494900 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:43.495470 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.496017 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:43.496914 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.497098 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:43.503704 (+ 6606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:41037: BlockingRecv error: recv got EOF from 127.2.164.65:41037 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
Feb 02 15:04:43 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:43 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:43.678092 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:43.672726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:40985)
0202 15:04:43.673098 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:43.673115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:43.673268 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:43.673561 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:43.673564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:43.673796 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:43.673996 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:43.674003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.674758 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:43.674765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:43.675154 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:43.675161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.675314 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:43.675911 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:43.675925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:43.677912 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
W20260202 15:04:43.678189 12566 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:43.672896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:40985 (local address 127.2.164.126:42593)
0202 15:04:43.673099 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:43.673105 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:43.673181 (+ 76us) server_negotiation.cc:408] Connection header received
0202 15:04:43.673382 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:43.673386 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:43.673444 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:43.673542 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:43.674130 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.674643 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:43.675298 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.675520 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:43.678056 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:40985: BlockingRecv error: recv got EOF from 127.2.164.66:40985 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Feb 02 15:04:43 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:43 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:43.858893 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:43.852935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:54103)
0202 15:04:43.853272 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:43.853287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:43.853416 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:43.853857 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:43.853861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:43.854107 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:43.854393 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:43.854404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.855514 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:43.855517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:43.855908 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:43.855916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.856025 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:43.856802 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:43.856818 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:43.858774 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":148,"threads_started":1}
W20260202 15:04:43.859077 12566 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:43.853132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:54103 (local address 127.2.164.126:42593)
0202 15:04:43.853304 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:43.853309 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:43.853382 (+ 73us) server_negotiation.cc:408] Connection header received
0202 15:04:43.853542 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:43.853546 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:43.853611 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:43.853717 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:43.854642 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.855370 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:43.856044 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:43.856192 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:43.858950 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:54103: BlockingRecv error: recv got EOF from 127.2.164.67:54103 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Feb 02 15:04:44 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:44 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:44.509671 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:44.504683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:52261)
0202 15:04:44.504964 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:44.504978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:44.505124 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:44.505441 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:44.505444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:44.505629 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:44.505853 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:44.505859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.506597 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:44.506599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:44.506990 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:44.506996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.507083 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:44.507632 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:44.507645 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:44.509507 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
W20260202 15:04:44.509855 12570 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:44.504760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:52261 (local address 127.2.164.126:42593)
0202 15:04:44.505076 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:44.505079 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:44.505093 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:44.505218 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:44.505220 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:44.505265 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:44.505354 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:44.505971 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.506488 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:44.507091 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.507231 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:44.509674 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:52261: BlockingRecv error: recv got EOF from 127.2.164.65:52261 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":134,"threads_started":1}
Feb 02 15:04:44 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:44 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:44.684378 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:44.679023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:58259)
0202 15:04:44.679342 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:44.679363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:44.679508 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:44.679801 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:44.679804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:44.680086 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:44.680285 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:44.680291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.681123 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:44.681128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:44.681513 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:44.681520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.681659 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:44.682214 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:44.682227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:44.684230 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
W20260202 15:04:44.684551 12570 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:44.679110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:58259 (local address 127.2.164.126:42593)
0202 15:04:44.679291 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:44.679297 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:44.679456 (+ 159us) server_negotiation.cc:408] Connection header received
0202 15:04:44.679617 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:44.679631 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:44.679688 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:44.679775 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:44.680440 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.680963 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:44.681642 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.681855 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:44.684388 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:58259: BlockingRecv error: recv got EOF from 127.2.164.66:58259 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Feb 02 15:04:44 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:44 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:44.865572 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:44.859869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:45973)
0202 15:04:44.860436 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:44.860449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:44.860551 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:44.860843 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:44.860845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:44.861056 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:44.861283 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:44.861294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.862387 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:44.862391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:44.862780 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:44.862788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.862890 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:44.863488 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:44.863502 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:44.865433 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":249,"threads_started":1}
W20260202 15:04:44.865726 12570 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:44.860019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45973 (local address 127.2.164.126:42593)
0202 15:04:44.860264 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:44.860268 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:44.860539 (+ 271us) server_negotiation.cc:408] Connection header received
0202 15:04:44.860631 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:44.860633 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:44.860722 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:44.860811 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:44.861476 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.862165 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:44.862920 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:44.863076 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:44.865594 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:45973: BlockingRecv error: recv got EOF from 127.2.164.67:45973 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Feb 02 15:04:45 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:45 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:45.515661 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:45.510426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:44843)
0202 15:04:45.510716 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:45.510734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:45.510852 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:45.511105 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:45.511108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:45.511313 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:45.511518 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:45.511524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.512262 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:45.512265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:45.512722 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:45.512731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.512843 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:45.513516 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:45.513535 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:45.515545 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
W20260202 15:04:45.515851 12574 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:45.510585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:44843 (local address 127.2.164.126:42593)
0202 15:04:45.510849 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:45.510853 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:45.510867 (+ 14us) server_negotiation.cc:408] Connection header received
0202 15:04:45.510928 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:45.510931 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:45.510976 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:45.511055 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:45.511644 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.512151 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:45.512838 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.513102 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:45.515704 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:44843: BlockingRecv error: recv got EOF from 127.2.164.65:44843 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
Feb 02 15:04:45 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:45 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:45.690732 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:45.685202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:58215)
0202 15:04:45.685535 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:45.685552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:45.685688 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:45.685936 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:45.685939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:45.686145 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:45.686374 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:45.686381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.687104 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:45.687107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:45.687521 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:45.687538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.687640 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:45.688261 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:45.688277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:45.690534 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":93,"threads_started":1}
W20260202 15:04:45.690788 12574 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:45.685309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:58215 (local address 127.2.164.126:42593)
0202 15:04:45.685510 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:45.685514 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:45.685636 (+ 122us) server_negotiation.cc:408] Connection header received
0202 15:04:45.685776 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:45.685778 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:45.685824 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:45.685893 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:45.686494 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.686997 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:45.687634 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.687763 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:45.690671 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:58215: BlockingRecv error: recv got EOF from 127.2.164.66:58215 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Feb 02 15:04:45 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:45 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:45.871793 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:45.866716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:33113)
0202 15:04:45.867040 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:45.867056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:45.867178 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:45.867482 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:45.867486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:45.867708 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:45.867926 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:45.867935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.868756 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:45.868759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:45.869166 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:45.869173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.869267 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:45.869875 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:45.869889 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:45.871680 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
W20260202 15:04:45.871948 12574 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:45.866833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:33113 (local address 127.2.164.126:42593)
0202 15:04:45.866978 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:45.866983 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:45.867163 (+ 180us) server_negotiation.cc:408] Connection header received
0202 15:04:45.867277 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:45.867280 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:45.867350 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:45.867456 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:45.868069 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.868621 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:45.869256 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:45.869393 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:45.871814 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:33113: BlockingRecv error: recv got EOF from 127.2.164.67:33113 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Feb 02 15:04:46 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:46 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:46.522682 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:46.516753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:43167)
0202 15:04:46.517139 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:46.517152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:46.517251 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:46.517684 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:46.517688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:46.517970 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:46.518208 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:46.518216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.519466 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:46.519470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:46.519857 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:46.519863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.519966 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:46.520567 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:46.520581 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:46.522487 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
W20260202 15:04:46.522746 12578 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:46.516803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:43167 (local address 127.2.164.126:42593)
0202 15:04:46.517232 (+ 429us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:46.517237 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:46.517257 (+ 20us) server_negotiation.cc:408] Connection header received
0202 15:04:46.517354 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:46.517357 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:46.517405 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:46.517636 (+ 231us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:46.518797 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.519320 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:46.520020 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.520168 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:46.522623 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:43167: BlockingRecv error: recv got EOF from 127.2.164.65:43167 (error 108)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":69,"threads_started":1}
Feb 02 15:04:46 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:46 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:46.697175 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:46.691683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:44285)
0202 15:04:46.692082 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:46.692095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:46.692205 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:46.692456 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:46.692459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:46.692651 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:46.692850 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:46.692855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.693663 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:46.693669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:46.694136 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:46.694145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.694281 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:46.694924 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:46.694946 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:46.697062 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":89,"threads_started":1}
W20260202 15:04:46.697381 12578 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:46.691827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:44285 (local address 127.2.164.126:42593)
0202 15:04:46.691964 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:46.691968 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:46.692183 (+ 215us) server_negotiation.cc:408] Connection header received
0202 15:04:46.692291 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:46.692294 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:46.692344 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:46.692418 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:46.692955 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.693518 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:46.694262 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.694462 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:46.697197 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:44285: BlockingRecv error: recv got EOF from 127.2.164.66:44285 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Feb 02 15:04:46 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:46 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:46.877492 12580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:46.872649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:54693)
0202 15:04:46.872963 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:46.872976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:46.873094 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:46.873340 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:46.873343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:46.873534 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:46.873734 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:46.873740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.874417 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:46.874419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:46.874800 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:46.874807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.874897 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:46.875424 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:46.875437 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:46.877298 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
W20260202 15:04:46.877513 12578 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:46.872687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:54693 (local address 127.2.164.126:42593)
0202 15:04:46.872845 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:46.872849 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:46.873069 (+ 220us) server_negotiation.cc:408] Connection header received
0202 15:04:46.873169 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:46.873171 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:46.873219 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:46.873284 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:46.873837 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.874319 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:46.874906 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:46.875034 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:46.877415 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:54693: BlockingRecv error: recv got EOF from 127.2.164.67:54693 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Feb 02 15:04:47 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:47 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:47.528963 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:47.523767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:54181)
0202 15:04:47.524083 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:47.524119 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:47.524251 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:47.524575 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:47.524578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:47.524859 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:47.525162 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:47.525172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.525921 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:47.525923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:47.526297 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:47.526303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.526392 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:47.527000 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:47.527013 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:47.528845 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
W20260202 15:04:47.529184 12582 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:47.523857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:54181 (local address 127.2.164.126:42593)
0202 15:04:47.524139 (+ 282us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:47.524149 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:47.524203 (+ 54us) server_negotiation.cc:408] Connection header received
0202 15:04:47.524372 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:47.524378 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:47.524440 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:47.524538 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:47.525271 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.525823 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:47.526394 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.526540 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:47.528989 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:54181: BlockingRecv error: recv got EOF from 127.2.164.65:54181 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
Feb 02 15:04:47 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:47 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:47.704173 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:47.698105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:38179)
0202 15:04:47.698453 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:47.698467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:47.698570 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:47.698848 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:47.698850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:47.699033 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:47.699229 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:47.699235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.700236 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:47.700239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:47.700689 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:47.700697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.700799 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:47.701477 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:47.701492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:47.704052 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
W20260202 15:04:47.704412 12582 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:47.698314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:38179 (local address 127.2.164.126:42593)
0202 15:04:47.698518 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:47.698522 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:47.698540 (+ 18us) server_negotiation.cc:408] Connection header received
0202 15:04:47.698675 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:47.698678 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:47.698729 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:47.698812 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:47.699474 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.700067 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:47.700978 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.701248 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:47.704227 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:38179: BlockingRecv error: recv got EOF from 127.2.164.66:38179 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Feb 02 15:04:47 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:47 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:47.883147 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:47.878389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:35427)
0202 15:04:47.878699 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:47.878713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:47.878816 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:47.879107 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:47.879110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:47.879302 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:47.879505 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:47.879511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.880236 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:47.880240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:47.880632 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:47.880639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.880743 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:47.881263 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:47.881276 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:47.883029 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
W20260202 15:04:47.883307 12582 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:47.878462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:35427 (local address 127.2.164.126:42593)
0202 15:04:47.878606 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:47.878610 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:47.878782 (+ 172us) server_negotiation.cc:408] Connection header received
0202 15:04:47.878900 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:47.878902 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:47.878950 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:47.879023 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:47.879614 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.880124 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:47.880733 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:47.880856 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:47.883152 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:35427: BlockingRecv error: recv got EOF from 127.2.164.67:35427 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Feb 02 15:04:48 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:48 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:48.538403 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:48.529876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:52285)
0202 15:04:48.530210 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:48.530229 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:48.530339 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:48.530604 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:48.530606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:48.530861 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:48.531128 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:48.531140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.531879 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:48.531882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:48.532256 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:48.532262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.532354 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:48.535896 (+ 3542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:48.535912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:48.538230 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
W20260202 15:04:48.538507 12586 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:48.529960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:52285 (local address 127.2.164.126:42593)
0202 15:04:48.530312 (+ 352us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:48.530316 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:48.530329 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:48.530441 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:48.530444 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:48.530489 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:48.530566 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:48.531244 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.531767 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:48.532354 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.532496 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:48.538365 (+ 5869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:52285: BlockingRecv error: recv got EOF from 127.2.164.65:52285 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":97,"threads_started":1}
Feb 02 15:04:48 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:48 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:48.710398 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:48.705163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:45741)
0202 15:04:48.705539 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:48.705552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:48.705708 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:48.706162 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:48.706166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:48.706422 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:48.706630 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:48.706636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.707387 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:48.707390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:48.707819 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:48.707827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.707929 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:48.708475 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:48.708489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:48.710286 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":87,"threads_started":1}
W20260202 15:04:48.710541 12586 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:48.705231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45741 (local address 127.2.164.126:42593)
0202 15:04:48.705407 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:48.705417 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:48.705634 (+ 217us) server_negotiation.cc:408] Connection header received
0202 15:04:48.705821 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:48.705824 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:48.706002 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:48.706101 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:48.706745 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.707244 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:48.707937 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.708073 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:48.710420 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:45741: BlockingRecv error: recv got EOF from 127.2.164.66:45741 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Feb 02 15:04:48 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:48 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:48.888810 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:48.884016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.67:41023)
0202 15:04:48.884294 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:48.884308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:48.884415 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:48.884665 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:48.884667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:48.884850 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:48.885122 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:48.885129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.885823 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:48.885826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:48.886213 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:48.886220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.886312 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:48.886872 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:48.886885 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:48.888691 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
W20260202 15:04:48.889065 12586 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:48.884155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:41023 (local address 127.2.164.126:42593)
0202 15:04:48.884316 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:48.884319 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:48.884377 (+ 58us) server_negotiation.cc:408] Connection header received
0202 15:04:48.884509 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:48.884512 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:48.884558 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:48.884628 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:48.885222 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.885715 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:48.886312 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:48.886434 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:48.888873 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:41023: BlockingRecv error: recv got EOF from 127.2.164.67:41023 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Feb 02 15:04:49 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:49 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:49.544448 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:49.539392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.65:34095)
0202 15:04:49.539726 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:49.539740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:49.539830 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:49.540199 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:49.540202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:49.540435 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:49.540634 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:49.540641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.541403 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:49.541406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:49.541792 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:49.541800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.541905 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:49.542443 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:49.542456 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:49.544269 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":135,"threads_started":1}
W20260202 15:04:49.544510 12590 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:49.539551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34095 (local address 127.2.164.126:42593)
0202 15:04:49.539787 (+ 236us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:49.539790 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:49.539803 (+ 13us) server_negotiation.cc:408] Connection header received
0202 15:04:49.539947 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:49.539951 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:49.540002 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:49.540095 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:49.540747 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.541280 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:49.541911 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.542046 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:49.544392 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:34095: BlockingRecv error: recv got EOF from 127.2.164.65:34095 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
Feb 02 15:04:49 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Feb 02 15:04:49 dist-test-slave-6zzp krb5kdc[11522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260202 15:04:49.716566 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:49.711486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42593 (local address 127.2.164.66:60031)
0202 15:04:49.711937 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:49.711954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:49.712066 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:49.712346 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:49.712349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:49.712611 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:49.712838 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:49.712845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.713598 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:49.713600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:49.713980 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:49.713987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.714074 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:49.714591 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:49.714604 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:49.716395 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:42593: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":89,"threads_started":1}
W20260202 15:04:49.716621 12590 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:49.711597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60031 (local address 127.2.164.126:42593)
0202 15:04:49.711814 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:49.711819 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:49.712045 (+ 226us) server_negotiation.cc:408] Connection header received
0202 15:04:49.712155 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:49.712159 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:49.712222 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:49.712283 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:49.712948 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.713499 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:49.714073 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:49.714196 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:49.716517 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:60031: BlockingRecv error: recv got EOF from 127.2.164.66:60031 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260202 15:04:49.793593 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12114
I20260202 15:04:49.799538 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12266
I20260202 15:04:49.805238 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12399
I20260202 15:04:49.810542 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12044
2026-02-02T15:04:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19017 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-02-02T15:04:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:49Z Disabled control of system clock
I20260202 15:04:49.840780 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38513
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260202 15:04:49.953627 12595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:49.953871 12595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:49.953917 12595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:49.957309 12595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:49.957376 12595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:49.957393 12595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:49.957412 12595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:49.957429 12595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:49.961766 12595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:49.962915 12595 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:49.964032 12595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:49.970341 12601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:49.970395 12600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:49.970592 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:49.970450 12595 server_base.cc:1060] running on GCE node
I20260202 15:04:49.971108 12595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:49.972055 12595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:49.973238 12595 hybrid_clock.cc:648] HybridClock initialized: now 1770044689973213 us; error 52 us; skew 500 ppm
I20260202 15:04:49.975190 12595 webserver.cc:492] Webserver started at http://127.2.164.126:40109/ using document root <none> and password file <none>
I20260202 15:04:49.975841 12595 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:49.975931 12595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:49.976182 12595 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:49.978015 12595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "147bcb61c44b47868c9e8b6360dc04b9"
format_stamp: "Formatted at 2026-02-02 15:04:49 on dist-test-slave-6zzp"
server_key: "b468a3dee4a04eb9027d8aff9e9a32de"
server_key_iv: "ea6943521564221465bec047f551a26f"
server_key_version: "encryptionkey@0"
I20260202 15:04:49.978531 12595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "147bcb61c44b47868c9e8b6360dc04b9"
format_stamp: "Formatted at 2026-02-02 15:04:49 on dist-test-slave-6zzp"
server_key: "b468a3dee4a04eb9027d8aff9e9a32de"
server_key_iv: "ea6943521564221465bec047f551a26f"
server_key_version: "encryptionkey@0"
I20260202 15:04:49.982420 12595 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260202 15:04:49.985495 12609 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:49.986639 12595 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260202 15:04:49.986753 12595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "147bcb61c44b47868c9e8b6360dc04b9"
format_stamp: "Formatted at 2026-02-02 15:04:49 on dist-test-slave-6zzp"
server_key: "b468a3dee4a04eb9027d8aff9e9a32de"
server_key_iv: "ea6943521564221465bec047f551a26f"
server_key_version: "encryptionkey@0"
I20260202 15:04:49.986842 12595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:50.010627 12595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:50.011371 12595 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:50.011647 12595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:50.019883 12595 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38513
I20260202 15:04:50.019891 12661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:38513 every 8 connection(s)
I20260202 15:04:50.021085 12595 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:50.023962 12662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:50.026542 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12595
I20260202 15:04:50.026647 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:50.026937 2705 external_mini_cluster.cc:1468] Setting key 9e4289f4ce8a64932857a0d5b4b018f4
I20260202 15:04:50.030104 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9: Bootstrap starting.
I20260202 15:04:50.032305 12662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:50.033215 12662 log.cc:826] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:50.034955 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.028279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.2.164.126:38513)
0202 15:04:50.028905 (+ 626us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:50.028914 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:50.028947 (+ 33us) server_negotiation.cc:408] Connection header received
0202 15:04:50.029524 (+ 577us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:50.029541 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:50.029897 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:50.030229 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:50.030747 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.031479 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.032182 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.032484 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.033092 (+ 608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:50.033111 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:50.033121 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:50.033627 (+ 506us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:50.033681 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:50.033701 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:50.033814 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:50.034015 (+ 201us) server_negotiation.cc:300] Negotiation successful
0202 15:04:50.034248 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260202 15:04:50.035323 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9: No bootstrap required, opened a new log
I20260202 15:04:50.038235 12662 raft_consensus.cc:359] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147bcb61c44b47868c9e8b6360dc04b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38513 } }
I20260202 15:04:50.038486 12662 raft_consensus.cc:385] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:50.038568 12662 raft_consensus.cc:740] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147bcb61c44b47868c9e8b6360dc04b9, State: Initialized, Role: FOLLOWER
I20260202 15:04:50.038982 12662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147bcb61c44b47868c9e8b6360dc04b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38513 } }
I20260202 15:04:50.039130 12662 raft_consensus.cc:399] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:50.039203 12662 raft_consensus.cc:493] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:50.039320 12662 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:50.040154 12662 raft_consensus.cc:515] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147bcb61c44b47868c9e8b6360dc04b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38513 } }
I20260202 15:04:50.040491 12662 leader_election.cc:304] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147bcb61c44b47868c9e8b6360dc04b9; no voters:
I20260202 15:04:50.040800 12662 leader_election.cc:290] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:50.040988 12667 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:50.041232 12667 raft_consensus.cc:697] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [term 1 LEADER]: Becoming Leader. State: Replica: 147bcb61c44b47868c9e8b6360dc04b9, State: Running, Role: LEADER
I20260202 15:04:50.041549 12667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147bcb61c44b47868c9e8b6360dc04b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38513 } }
I20260202 15:04:50.042053 12662 sys_catalog.cc:565] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:50.043355 12669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147bcb61c44b47868c9e8b6360dc04b9. Latest consensus state: current_term: 1 leader_uuid: "147bcb61c44b47868c9e8b6360dc04b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147bcb61c44b47868c9e8b6360dc04b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38513 } } }
I20260202 15:04:50.043548 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147bcb61c44b47868c9e8b6360dc04b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147bcb61c44b47868c9e8b6360dc04b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38513 } } }
I20260202 15:04:50.043601 12669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:50.043654 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:50.044019 12676 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:50.047175 12676 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:50.052482 12676 catalog_manager.cc:1357] Generated new cluster ID: 48a032d35cf74fb0af4d1939b266e100
I20260202 15:04:50.052582 12676 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:50.067880 12676 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:50.068813 12676 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:50.079450 12676 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 147bcb61c44b47868c9e8b6360dc04b9: Generated new TSK 0
I20260202 15:04:50.080215 12676 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:04:50.083309 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38513
--builtin_ntp_servers=127.2.164.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260202 15:04:50.191461 12686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:50.191715 12686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:50.191794 12686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:50.195024 12686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:50.195113 12686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:50.195216 12686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:50.199143 12686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:50.200285 12686 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:50.201458 12686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:50.208213 12691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:50.208261 12694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:50.208384 12686 server_base.cc:1060] running on GCE node
W20260202 15:04:50.208518 12692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:50.208909 12686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:50.209527 12686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:50.210774 12686 hybrid_clock.cc:648] HybridClock initialized: now 1770044690210765 us; error 33 us; skew 500 ppm
I20260202 15:04:50.212582 12686 webserver.cc:492] Webserver started at http://127.2.164.65:44455/ using document root <none> and password file <none>
I20260202 15:04:50.213230 12686 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:50.213335 12686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:50.213562 12686 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:50.215231 12686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "fb686720d86d4a4ba98091ff713feedf"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "5a007b45cce8a15f5eb773fdf1950b9d"
server_key_iv: "29a8270aa2d363a2b4ccb625924d4d72"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.215718 12686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "fb686720d86d4a4ba98091ff713feedf"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "5a007b45cce8a15f5eb773fdf1950b9d"
server_key_iv: "29a8270aa2d363a2b4ccb625924d4d72"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.219197 12686 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:04:50.221382 12700 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.222422 12686 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:50.222548 12686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "fb686720d86d4a4ba98091ff713feedf"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "5a007b45cce8a15f5eb773fdf1950b9d"
server_key_iv: "29a8270aa2d363a2b4ccb625924d4d72"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.222648 12686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:50.247596 12686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:50.248324 12686 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:50.248545 12686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:50.249220 12686 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:50.250185 12686 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:50.250259 12686 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.250344 12686 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:50.250398 12686 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.259732 12686 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:46551
I20260202 15:04:50.259801 12813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:46551 every 8 connection(s)
I20260202 15:04:50.260704 12686 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:50.269155 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.262626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50577 (local address 127.2.164.126:38513)
0202 15:04:50.262807 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:50.262812 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:50.263897 (+ 1085us) server_negotiation.cc:408] Connection header received
0202 15:04:50.264761 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:50.264765 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:50.264817 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:50.264894 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:50.265900 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.266392 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.267004 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.267135 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.268634 (+ 1499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:50.268692 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:50.268698 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:50.268712 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:50.268739 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:50.268747 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:50.268810 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:50.268937 (+ 127us) server_negotiation.cc:300] Negotiation successful
0202 15:04:50.268985 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260202 15:04:50.269876 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12686
I20260202 15:04:50.270000 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:50.270013 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.262925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38513 (local address 127.2.164.65:50577)
0202 15:04:50.263739 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:50.263776 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:50.264518 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:50.265103 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:50.265111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:50.265186 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:50.265763 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:50.265775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.266506 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.266508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:50.266905 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:50.266911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.267077 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.268361 (+ 1284us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:04:50.268363 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:50.268423 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:04:50.268493 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:50.268826 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:50.268832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:50.268835 (+ 3us) client_negotiation.cc:769] Sending connection context
0202 15:04:50.268942 (+ 107us) client_negotiation.cc:240] Negotiation successful
0202 15:04:50.269176 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260202 15:04:50.270298 2705 external_mini_cluster.cc:1468] Setting key 702a516fe6c28b75749d59d7dbbf21b7
I20260202 15:04:50.271286 12814 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38513
I20260202 15:04:50.271550 12814 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:50.272089 12814 heartbeater.cc:507] Master 127.2.164.126:38513 requested a full tablet report, sending...
I20260202 15:04:50.272148 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38513
--builtin_ntp_servers=127.2.164.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260202 15:04:50.273720 12626 ts_manager.cc:194] Registered new tserver with Master: fb686720d86d4a4ba98091ff713feedf (127.2.164.65:46551)
I20260202 15:04:50.274951 12626 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.164.65:50577
W20260202 15:04:50.380219 12817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:50.380450 12817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:50.380496 12817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:50.383895 12817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:50.383956 12817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:50.384027 12817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:50.387881 12817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:50.388969 12817 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:50.390144 12817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:50.396860 12825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:50.396874 12822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:50.396984 12823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:50.397192 12817 server_base.cc:1060] running on GCE node
I20260202 15:04:50.397656 12817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:50.398207 12817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:50.399405 12817 hybrid_clock.cc:648] HybridClock initialized: now 1770044690399389 us; error 33 us; skew 500 ppm
I20260202 15:04:50.401217 12817 webserver.cc:492] Webserver started at http://127.2.164.66:43713/ using document root <none> and password file <none>
I20260202 15:04:50.401798 12817 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:50.401882 12817 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:50.402102 12817 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:50.403771 12817 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "6214910d9efa4209b40f34fe7bf033fb"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "6d4a0d93223999e5d7580d3530d26c44"
server_key_iv: "78ee0a6157d8c975ecc6a09917fafadb"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.404254 12817 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "6214910d9efa4209b40f34fe7bf033fb"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "6d4a0d93223999e5d7580d3530d26c44"
server_key_iv: "78ee0a6157d8c975ecc6a09917fafadb"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.407708 12817 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260202 15:04:50.410063 12831 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.411087 12817 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:50.411202 12817 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "6214910d9efa4209b40f34fe7bf033fb"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "6d4a0d93223999e5d7580d3530d26c44"
server_key_iv: "78ee0a6157d8c975ecc6a09917fafadb"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.411417 12817 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:50.429380 12817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:50.430060 12817 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:50.430250 12817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:50.430821 12817 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:50.431800 12817 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:50.431847 12817 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.431914 12817 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:50.431950 12817 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.440910 12817 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:40547
I20260202 15:04:50.440938 12944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:40547 every 8 connection(s)
I20260202 15:04:50.441893 12817 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:50.449994 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12817
I20260202 15:04:50.450169 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:50.450312 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.443730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:56505 (local address 127.2.164.126:38513)
0202 15:04:50.443877 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:50.443881 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:50.444758 (+ 877us) server_negotiation.cc:408] Connection header received
0202 15:04:50.445665 (+ 907us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:50.445669 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:50.445732 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:50.445842 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:50.446845 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.447344 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.448009 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.448196 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.449710 (+ 1514us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:50.449740 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:50.449743 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:50.449766 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:50.449790 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:50.449798 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:50.449881 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:50.450079 (+ 198us) server_negotiation.cc:300] Negotiation successful
0202 15:04:50.450157 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260202 15:04:50.450474 2705 external_mini_cluster.cc:1468] Setting key 476027b90813b3cffd72271f1af8466e
I20260202 15:04:50.451200 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.443997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38513 (local address 127.2.164.66:56505)
0202 15:04:50.444609 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:50.444645 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:50.445458 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:50.446004 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:50.446012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:50.446094 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:50.446677 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:50.446689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.447477 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.447482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:50.447889 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:50.447896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.448149 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.449453 (+ 1304us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:04:50.449455 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:50.449495 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:04:50.449559 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:50.449906 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:50.449921 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:50.449925 (+ 4us) client_negotiation.cc:769] Sending connection context
0202 15:04:50.450083 (+ 158us) client_negotiation.cc:240] Negotiation successful
0202 15:04:50.450382 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260202 15:04:50.452620 12945 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38513
I20260202 15:04:50.452827 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38513
--builtin_ntp_servers=127.2.164.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260202 15:04:50.452901 12945 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:50.453534 12945 heartbeater.cc:507] Master 127.2.164.126:38513 requested a full tablet report, sending...
I20260202 15:04:50.454784 12625 ts_manager.cc:194] Registered new tserver with Master: 6214910d9efa4209b40f34fe7bf033fb (127.2.164.66:40547)
I20260202 15:04:50.455406 12625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.164.66:56505
W20260202 15:04:50.567386 12948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:50.567606 12948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:50.567651 12948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:50.571019 12948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:50.571079 12948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:50.571187 12948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:50.575066 12948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:50.576285 12948 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:50.577629 12948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:50.584355 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:50.584410 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:50.584461 12948 server_base.cc:1060] running on GCE node
W20260202 15:04:50.584726 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:50.585156 12948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:50.585728 12948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:50.586910 12948 hybrid_clock.cc:648] HybridClock initialized: now 1770044690586890 us; error 44 us; skew 500 ppm
I20260202 15:04:50.588702 12948 webserver.cc:492] Webserver started at http://127.2.164.67:36397/ using document root <none> and password file <none>
I20260202 15:04:50.589339 12948 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:50.589421 12948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:50.589648 12948 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:50.591298 12948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "64cbd19b3ec640a6a345193a4262fc3e"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "97b9c2a57cb6df788ea5a96b8512c517"
server_key_iv: "eaa09cf6fba47325cd71f89b715eeded"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.591794 12948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "64cbd19b3ec640a6a345193a4262fc3e"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "97b9c2a57cb6df788ea5a96b8512c517"
server_key_iv: "eaa09cf6fba47325cd71f89b715eeded"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.595192 12948 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260202 15:04:50.597514 12962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.598526 12948 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260202 15:04:50.598655 12948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "64cbd19b3ec640a6a345193a4262fc3e"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "97b9c2a57cb6df788ea5a96b8512c517"
server_key_iv: "eaa09cf6fba47325cd71f89b715eeded"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.598757 12948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:50.614475 12948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:50.615149 12948 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:50.615347 12948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:50.615950 12948 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:50.616882 12948 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:50.616928 12948 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.616968 12948 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:50.616983 12948 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.626500 12948 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:46381
I20260202 15:04:50.626524 13075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:46381 every 8 connection(s)
I20260202 15:04:50.627518 12948 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:50.630354 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 12948
I20260202 15:04:50.630487 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:50.630744 2705 external_mini_cluster.cc:1468] Setting key bd93e88f569cf552a48f8341af38ef3d
I20260202 15:04:50.636071 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.629318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59051 (local address 127.2.164.126:38513)
0202 15:04:50.629470 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:50.629474 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:50.630438 (+ 964us) server_negotiation.cc:408] Connection header received
0202 15:04:50.631502 (+ 1064us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:50.631507 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:50.631565 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:50.631659 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:50.632850 (+ 1191us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.633459 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.634150 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.634316 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.635573 (+ 1257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:50.635587 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:50.635590 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:50.635601 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:50.635621 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:50.635629 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:50.635714 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:50.635910 (+ 196us) server_negotiation.cc:300] Negotiation successful
0202 15:04:50.635962 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260202 15:04:50.636751 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.629598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38513 (local address 127.2.164.67:59051)
0202 15:04:50.630255 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:50.630301 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:50.631250 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:50.631869 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:50.631879 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:50.631977 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:50.632653 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:50.632665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.633600 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:50.633604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:50.634018 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:50.634025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:50.634255 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:50.635349 (+ 1094us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:04:50.635351 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:50.635391 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:04:50.635443 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:50.635758 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:50.635765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:50.635769 (+ 4us) client_negotiation.cc:769] Sending connection context
0202 15:04:50.635894 (+ 125us) client_negotiation.cc:240] Negotiation successful
0202 15:04:50.636091 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260202 15:04:50.637846 13076 heartbeater.cc:344] Connected to a master server at 127.2.164.126:38513
I20260202 15:04:50.638111 13076 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:50.638686 13076 heartbeater.cc:507] Master 127.2.164.126:38513 requested a full tablet report, sending...
I20260202 15:04:50.639752 12625 ts_manager.cc:194] Registered new tserver with Master: 64cbd19b3ec640a6a345193a4262fc3e (127.2.164.67:46381)
I20260202 15:04:50.640332 12625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.2.164.67:59051
I20260202 15:04:50.644907 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260202 15:04:50.647697 12665 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:50.646819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.2.164.126:38513)
0202 15:04:50.646945 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:50.646949 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:50.647134 (+ 185us) server_negotiation.cc:408] Connection header received
0202 15:04:50.647207 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:50.647210 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:50.647257 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:50.647336 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:50.647525 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49116: BlockingRecv error: recv got EOF from 127.0.0.1:49116 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:04:50.647980 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12686
I20260202 15:04:50.654755 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12817
I20260202 15:04:50.660952 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12948
I20260202 15:04:50.666853 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 12595
2026-02-02T15:04:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (855 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-02-02T15:04:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:50Z Disabled control of system clock
I20260202 15:04:50.695534 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44675
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260202 15:04:50.803081 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:50.803318 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:50.803366 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:50.806660 13087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:50.806723 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:50.806739 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:50.806758 13087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:50.806777 13087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:50.810699 13087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:50.811772 13087 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:50.812880 13087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:50.818697 13092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:50.818714 13093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:50.819126 13095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:50.819347 13087 server_base.cc:1060] running on GCE node
I20260202 15:04:50.819856 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:50.820804 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:50.821995 13087 hybrid_clock.cc:648] HybridClock initialized: now 1770044690822014 us; error 59 us; skew 500 ppm
I20260202 15:04:50.823833 13087 webserver.cc:492] Webserver started at http://127.2.164.126:46665/ using document root <none> and password file <none>
I20260202 15:04:50.824407 13087 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:50.824491 13087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:50.824715 13087 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:50.826474 13087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "d12380e479f345d9983cfb8bcb80fa91"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "eabc975c1ace50d017a6ea5a3d7a0a06"
server_key_iv: "f9d363b94a68c680fa241fc6e358e4f9"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.826978 13087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "d12380e479f345d9983cfb8bcb80fa91"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "eabc975c1ace50d017a6ea5a3d7a0a06"
server_key_iv: "f9d363b94a68c680fa241fc6e358e4f9"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.830431 13087 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:50.832692 13101 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:50.833765 13087 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:50.833900 13087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "d12380e479f345d9983cfb8bcb80fa91"
format_stamp: "Formatted at 2026-02-02 15:04:50 on dist-test-slave-6zzp"
server_key: "eabc975c1ace50d017a6ea5a3d7a0a06"
server_key_iv: "f9d363b94a68c680fa241fc6e358e4f9"
server_key_version: "encryptionkey@0"
I20260202 15:04:50.834003 13087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:50.838915 13087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:50.839471 13087 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:50.839649 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:50.846371 13087 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44675
I20260202 15:04:50.846377 13153 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:44675 every 8 connection(s)
I20260202 15:04:50.847368 13087 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:50.850219 13154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:50.850873 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13087
I20260202 15:04:50.851034 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:50.851291 2705 external_mini_cluster.cc:1468] Setting key c096bd7630e47afa3d8cc0701750202c
I20260202 15:04:50.856827 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:50.852524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.2.164.126:44675)
0202 15:04:50.853177 (+ 653us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:50.853186 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:50.853223 (+ 37us) server_negotiation.cc:408] Connection header received
0202 15:04:50.854209 (+ 986us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:50.854229 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:50.854570 (+ 341us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:50.854847 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:50.855689 (+ 842us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:50.855694 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:50.855702 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:50.855713 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:50.855757 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:50.855768 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:50.855882 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:50.855995 (+ 113us) server_negotiation.cc:300] Negotiation successful
0202 15:04:50.856253 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260202 15:04:50.857474 13154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91: Bootstrap starting.
I20260202 15:04:50.859853 13154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:50.860550 13154 log.cc:826] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:50.862622 13154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91: No bootstrap required, opened a new log
I20260202 15:04:50.865342 13154 raft_consensus.cc:359] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12380e479f345d9983cfb8bcb80fa91" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44675 } }
I20260202 15:04:50.865561 13154 raft_consensus.cc:385] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:50.865643 13154 raft_consensus.cc:740] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12380e479f345d9983cfb8bcb80fa91, State: Initialized, Role: FOLLOWER
I20260202 15:04:50.866082 13154 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12380e479f345d9983cfb8bcb80fa91" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44675 } }
I20260202 15:04:50.866240 13154 raft_consensus.cc:399] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:50.866314 13154 raft_consensus.cc:493] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:50.866429 13154 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:50.867322 13154 raft_consensus.cc:515] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12380e479f345d9983cfb8bcb80fa91" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44675 } }
I20260202 15:04:50.867677 13154 leader_election.cc:304] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d12380e479f345d9983cfb8bcb80fa91; no voters:
I20260202 15:04:50.868022 13154 leader_election.cc:290] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:50.868161 13159 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:50.868389 13159 raft_consensus.cc:697] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [term 1 LEADER]: Becoming Leader. State: Replica: d12380e479f345d9983cfb8bcb80fa91, State: Running, Role: LEADER
I20260202 15:04:50.868747 13159 consensus_queue.cc:237] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12380e479f345d9983cfb8bcb80fa91" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44675 } }
I20260202 15:04:50.869297 13154 sys_catalog.cc:565] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:50.870594 13160 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d12380e479f345d9983cfb8bcb80fa91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12380e479f345d9983cfb8bcb80fa91" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44675 } } }
I20260202 15:04:50.870728 13160 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:50.870586 13161 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d12380e479f345d9983cfb8bcb80fa91. Latest consensus state: current_term: 1 leader_uuid: "d12380e479f345d9983cfb8bcb80fa91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12380e479f345d9983cfb8bcb80fa91" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44675 } } }
I20260202 15:04:50.870805 13161 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:50.871120 13168 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:50.873971 13168 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:50.879833 13168 catalog_manager.cc:1357] Generated new cluster ID: 714e4ef652624851ac98a313e71f1502
I20260202 15:04:50.879916 13168 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:50.890810 13168 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:50.891151 13168 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:50.897940 13168 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d12380e479f345d9983cfb8bcb80fa91: Generated new TSK 0
I20260202 15:04:50.898682 13168 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260202 15:04:50.905541 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44675
--builtin_ntp_servers=127.2.164.84:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260202 15:04:51.018657 13178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:51.018869 13178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:51.018922 13178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:51.022440 13178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:51.022504 13178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:51.022615 13178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:51.026677 13178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:51.027863 13178 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:51.029103 13178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:51.035856 13186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:51.035854 13183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:51.036000 13178 server_base.cc:1060] running on GCE node
W20260202 15:04:51.036079 13184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:51.036437 13178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:51.037042 13178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:51.038239 13178 hybrid_clock.cc:648] HybridClock initialized: now 1770044691038222 us; error 39 us; skew 500 ppm
I20260202 15:04:51.040092 13178 webserver.cc:492] Webserver started at http://127.2.164.65:40715/ using document root <none> and password file <none>
I20260202 15:04:51.040719 13178 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:51.040779 13178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:51.040982 13178 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:51.042707 13178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "98c0d4ab720a4399aa7cf56f22e0c02a"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "80e9f6b57bf0baea3afc481721febbcd"
server_key_iv: "e46f7dd2e2ae575659f6660d9138e792"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.043185 13178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "98c0d4ab720a4399aa7cf56f22e0c02a"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "80e9f6b57bf0baea3afc481721febbcd"
server_key_iv: "e46f7dd2e2ae575659f6660d9138e792"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.046766 13178 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260202 15:04:51.049192 13192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.050249 13178 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:51.050365 13178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "98c0d4ab720a4399aa7cf56f22e0c02a"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "80e9f6b57bf0baea3afc481721febbcd"
server_key_iv: "e46f7dd2e2ae575659f6660d9138e792"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.050464 13178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:51.055183 13178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:51.055757 13178 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:51.055936 13178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:51.056473 13178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:51.057467 13178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:51.057514 13178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.057577 13178 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:51.057607 13178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.066913 13178 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:34933
I20260202 15:04:51.066934 13305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:34933 every 8 connection(s)
I20260202 15:04:51.067855 13178 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:51.071321 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13178
I20260202 15:04:51.071460 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:51.071724 2705 external_mini_cluster.cc:1468] Setting key aac3dc9f51da90c010d6623d0bd491e7
I20260202 15:04:51.073459 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:51.069625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:39483 (local address 127.2.164.126:44675)
0202 15:04:51.069804 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:51.069809 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:51.070749 (+ 940us) server_negotiation.cc:408] Connection header received
0202 15:04:51.071705 (+ 956us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:51.071708 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:51.071756 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:51.071861 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:51.072931 (+ 1070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:51.072935 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:51.072937 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:51.072949 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:51.072972 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:51.072988 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:51.073126 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:51.073286 (+ 160us) server_negotiation.cc:300] Negotiation successful
0202 15:04:51.073330 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260202 15:04:51.073742 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44675
--builtin_ntp_servers=127.2.164.84:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260202 15:04:51.074276 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:51.069917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44675 (local address 127.2.164.65:39483)
0202 15:04:51.070592 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:51.070628 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:51.071464 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:51.072012 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:51.072020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:51.072101 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:51.072710 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:04:51.072711 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:51.072745 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:04:51.072795 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:51.073126 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:51.073138 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:51.073145 (+ 7us) client_negotiation.cc:769] Sending connection context
0202 15:04:51.073350 (+ 205us) client_negotiation.cc:240] Negotiation successful
0202 15:04:51.073569 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260202 15:04:51.075309 13306 heartbeater.cc:344] Connected to a master server at 127.2.164.126:44675
I20260202 15:04:51.075585 13306 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:51.076035 13306 heartbeater.cc:507] Master 127.2.164.126:44675 requested a full tablet report, sending...
I20260202 15:04:51.077734 13118 ts_manager.cc:194] Registered new tserver with Master: 98c0d4ab720a4399aa7cf56f22e0c02a (127.2.164.65:34933)
W20260202 15:04:51.183671 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:51.183913 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:51.183967 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:51.187353 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:51.187418 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:51.187542 13309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:51.191421 13309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:51.192571 13309 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:51.193794 13309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:51.200366 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:51.200532 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:51.200471 13317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:51.200757 13309 server_base.cc:1060] running on GCE node
I20260202 15:04:51.201316 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:51.201905 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:51.203086 13309 hybrid_clock.cc:648] HybridClock initialized: now 1770044691203065 us; error 42 us; skew 500 ppm
I20260202 15:04:51.204975 13309 webserver.cc:492] Webserver started at http://127.2.164.66:37193/ using document root <none> and password file <none>
I20260202 15:04:51.205611 13309 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:51.205693 13309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:51.205914 13309 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:51.207577 13309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "e57465c6a6434c19a063b14cd6e97fdd"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "820f061c51af78ae1abf52d44e758dd5"
server_key_iv: "c794ed5d9433c043fc730e20c07d4fbb"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.208081 13309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "e57465c6a6434c19a063b14cd6e97fdd"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "820f061c51af78ae1abf52d44e758dd5"
server_key_iv: "c794ed5d9433c043fc730e20c07d4fbb"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.211505 13309 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260202 15:04:51.214308 13323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.215386 13309 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260202 15:04:51.215540 13309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "e57465c6a6434c19a063b14cd6e97fdd"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "820f061c51af78ae1abf52d44e758dd5"
server_key_iv: "c794ed5d9433c043fc730e20c07d4fbb"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.215653 13309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:51.220515 13309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:51.221151 13309 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:51.221320 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:51.221868 13309 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:51.222791 13309 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:51.222862 13309 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.222929 13309 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:51.222971 13309 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.232427 13309 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:37643
I20260202 15:04:51.232467 13436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:37643 every 8 connection(s)
I20260202 15:04:51.233479 13309 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:51.239106 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:51.235194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:59661 (local address 127.2.164.126:44675)
0202 15:04:51.235334 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:51.235338 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:51.236172 (+ 834us) server_negotiation.cc:408] Connection header received
0202 15:04:51.236920 (+ 748us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:51.236923 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:51.236964 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:51.237102 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:51.238466 (+ 1364us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:51.238472 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:51.238475 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:51.238496 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:51.238527 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:51.238542 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:51.238675 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:51.238913 (+ 238us) server_negotiation.cc:300] Negotiation successful
0202 15:04:51.238964 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:04:51.239859 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:51.235450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44675 (local address 127.2.164.66:59661)
0202 15:04:51.236037 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:51.236069 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:51.236748 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:51.237234 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:51.237241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:51.237322 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:51.238031 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:04:51.238035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:51.238080 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:04:51.238157 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:51.238675 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:51.238702 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:51.238710 (+ 8us) client_negotiation.cc:769] Sending connection context
0202 15:04:51.238981 (+ 271us) client_negotiation.cc:240] Negotiation successful
0202 15:04:51.239195 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260202 15:04:51.240268 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13309
I20260202 15:04:51.240403 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:51.240661 2705 external_mini_cluster.cc:1468] Setting key a8252c367b855284309578fe645fa7ff
I20260202 15:04:51.241035 13437 heartbeater.cc:344] Connected to a master server at 127.2.164.126:44675
I20260202 15:04:51.241317 13437 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:51.241806 13437 heartbeater.cc:507] Master 127.2.164.126:44675 requested a full tablet report, sending...
I20260202 15:04:51.242662 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44675
--builtin_ntp_servers=127.2.164.84:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260202 15:04:51.242878 13118 ts_manager.cc:194] Registered new tserver with Master: e57465c6a6434c19a063b14cd6e97fdd (127.2.164.66:37643)
W20260202 15:04:51.352648 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:51.352873 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:51.352926 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:51.356302 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:51.356380 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:51.356491 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:51.360236 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:51.361447 13440 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:51.362694 13440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:51.369781 13446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:51.369838 13445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:51.369825 13448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:51.370018 13440 server_base.cc:1060] running on GCE node
I20260202 15:04:51.370385 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:51.370973 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:51.372138 13440 hybrid_clock.cc:648] HybridClock initialized: now 1770044691372135 us; error 40 us; skew 500 ppm
I20260202 15:04:51.373864 13440 webserver.cc:492] Webserver started at http://127.2.164.67:46821/ using document root <none> and password file <none>
I20260202 15:04:51.374507 13440 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:51.374591 13440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:51.374809 13440 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:51.376502 13440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "23d3936cd5c042dc9f73332a8409a446"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "0afe3c07032a7eff090637c73eaef174"
server_key_iv: "22f2f4fefb73240f7e9c1947eb4872fd"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.377038 13440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "23d3936cd5c042dc9f73332a8409a446"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "0afe3c07032a7eff090637c73eaef174"
server_key_iv: "22f2f4fefb73240f7e9c1947eb4872fd"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.380710 13440 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260202 15:04:51.383111 13454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.384155 13440 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:51.384249 13440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "23d3936cd5c042dc9f73332a8409a446"
format_stamp: "Formatted at 2026-02-02 15:04:51 on dist-test-slave-6zzp"
server_key: "0afe3c07032a7eff090637c73eaef174"
server_key_iv: "22f2f4fefb73240f7e9c1947eb4872fd"
server_key_version: "encryptionkey@0"
I20260202 15:04:51.384387 13440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:51.389216 13440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:51.389796 13440 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:51.389973 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:51.390496 13440 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:51.391415 13440 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:51.391480 13440 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.391521 13440 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:51.391536 13440 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:51.402012 13440 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:41223
I20260202 15:04:51.402033 13567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:41223 every 8 connection(s)
I20260202 15:04:51.402962 13440 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:51.407982 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:51.404678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:48049 (local address 127.2.164.126:44675)
0202 15:04:51.404823 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:51.404827 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:51.405798 (+ 971us) server_negotiation.cc:408] Connection header received
0202 15:04:51.406533 (+ 735us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:51.406536 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:51.406576 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:51.406652 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:51.407581 (+ 929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:51.407585 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:51.407587 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0202 15:04:51.407598 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:51.407616 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0202 15:04:51.407624 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:51.407701 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:51.407840 (+ 139us) server_negotiation.cc:300] Negotiation successful
0202 15:04:51.407876 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260202 15:04:51.408684 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:51.404983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44675 (local address 127.2.164.67:48049)
0202 15:04:51.405642 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:51.405677 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:51.406365 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:51.406775 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:51.406782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:51.406860 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:51.407362 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0202 15:04:51.407363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:51.407395 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0202 15:04:51.407446 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:51.407702 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:51.407707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:51.407711 (+ 4us) client_negotiation.cc:769] Sending connection context
0202 15:04:51.407880 (+ 169us) client_negotiation.cc:240] Negotiation successful
0202 15:04:51.408040 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
I20260202 15:04:51.408792 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13440
I20260202 15:04:51.408946 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:51.409274 2705 external_mini_cluster.cc:1468] Setting key 20d4162d290054d5232c1ded1484db5e
I20260202 15:04:51.409761 13568 heartbeater.cc:344] Connected to a master server at 127.2.164.126:44675
I20260202 15:04:51.410035 13568 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:51.410519 13568 heartbeater.cc:507] Master 127.2.164.126:44675 requested a full tablet report, sending...
I20260202 15:04:51.411553 13118 ts_manager.cc:194] Registered new tserver with Master: 23d3936cd5c042dc9f73332a8409a446 (127.2.164.67:41223)
I20260202 15:04:51.423025 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260202 15:04:51.426045 13157 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:04:51.425087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.2.164.126:44675)
0202 15:04:51.425231 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:51.425237 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:51.425395 (+ 158us) server_negotiation.cc:408] Connection header received
0202 15:04:51.425553 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:51.425558 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:51.425605 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:51.425706 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:51.425906 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43138: BlockingRecv error: recv got EOF from 127.0.0.1:43138 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260202 15:04:51.426196 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13178
I20260202 15:04:51.432379 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13309
I20260202 15:04:51.438489 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13440
I20260202 15:04:51.443943 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13087
2026-02-02T15:04:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (777 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:51 dist-test-slave-6zzp krb5kdc[13578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:51 dist-test-slave-6zzp krb5kdc[13578](info): set up 2 sockets
Feb 02 15:04:51 dist-test-slave-6zzp krb5kdc[13578](info): commencing operation
krb5kdc: starting...
W20260202 15:04:53.462239 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:53 dist-test-slave-6zzp krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:53.608402 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35289
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:53.718402 13594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:53.718766 13594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:53.718888 13594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:53.722328 13594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:04:53.722455 13594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:53.722503 13594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:53.722610 13594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:04:53.722680 13594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:04:53.727165 13594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:53.728384 13594 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:53.729312 13594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:53.735198 13600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:53.735229 13599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:53.735266 13602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:53.735837 13594 server_base.cc:1060] running on GCE node
I20260202 15:04:53.736423 13594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:53.737618 13594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:53.738816 13594 hybrid_clock.cc:648] HybridClock initialized: now 1770044693738802 us; error 39 us; skew 500 ppm
Feb 02 15:04:53 dist-test-slave-6zzp krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:53.741698 13594 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:04:53.743126 13594 webserver.cc:492] Webserver started at http://127.2.164.126:41407/ using document root <none> and password file <none>
I20260202 15:04:53.743777 13594 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:53.743853 13594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:53.744055 13594 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:53.745908 13594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "370b0428319a4bf3822912921671eaf3"
format_stamp: "Formatted at 2026-02-02 15:04:53 on dist-test-slave-6zzp"
server_key: "9901f0461cd8eeb9be1cb4ea68ee9ada"
server_key_iv: "c1c7037bb8f8e53e185d91857c79228a"
server_key_version: "encryptionkey@0"
I20260202 15:04:53.746433 13594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "370b0428319a4bf3822912921671eaf3"
format_stamp: "Formatted at 2026-02-02 15:04:53 on dist-test-slave-6zzp"
server_key: "9901f0461cd8eeb9be1cb4ea68ee9ada"
server_key_iv: "c1c7037bb8f8e53e185d91857c79228a"
server_key_version: "encryptionkey@0"
I20260202 15:04:53.749955 13594 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:53.752130 13609 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:53.753216 13594 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:53.753342 13594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "370b0428319a4bf3822912921671eaf3"
format_stamp: "Formatted at 2026-02-02 15:04:53 on dist-test-slave-6zzp"
server_key: "9901f0461cd8eeb9be1cb4ea68ee9ada"
server_key_iv: "c1c7037bb8f8e53e185d91857c79228a"
server_key_version: "encryptionkey@0"
I20260202 15:04:53.753449 13594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:53.771538 13594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:53.777650 13594 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:53.777884 13594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:53.785593 13594 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35289
I20260202 15:04:53.785652 13661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:35289 every 8 connection(s)
I20260202 15:04:53.786651 13594 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:04:53.789479 13662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:53.794363 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13594
I20260202 15:04:53.794512 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:04:53.794754 2705 external_mini_cluster.cc:1468] Setting key b32bda6c36f2c49394369ec042c4b0f0
I20260202 15:04:53.795217 13662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: Bootstrap starting.
I20260202 15:04:53.797648 13662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:53.798391 13662 log.cc:826] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:53.800107 13662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: No bootstrap required, opened a new log
Feb 02 15:04:53 dist-test-slave-6zzp krb5kdc[13578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:53.803267 13662 raft_consensus.cc:359] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b0428319a4bf3822912921671eaf3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35289 } }
I20260202 15:04:53.803521 13662 raft_consensus.cc:385] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:53.803603 13662 raft_consensus.cc:740] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b0428319a4bf3822912921671eaf3, State: Initialized, Role: FOLLOWER
I20260202 15:04:53.804275 13662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b0428319a4bf3822912921671eaf3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35289 } }
I20260202 15:04:53.804450 13662 raft_consensus.cc:399] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:04:53.804558 13662 raft_consensus.cc:493] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:04:53.804685 13662 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:53.805651 13662 raft_consensus.cc:515] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b0428319a4bf3822912921671eaf3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35289 } }
I20260202 15:04:53.806028 13662 leader_election.cc:304] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 370b0428319a4bf3822912921671eaf3; no voters:
I20260202 15:04:53.806377 13662 leader_election.cc:290] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:04:53.806461 13667 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:53.806655 13667 raft_consensus.cc:697] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [term 1 LEADER]: Becoming Leader. State: Replica: 370b0428319a4bf3822912921671eaf3, State: Running, Role: LEADER
I20260202 15:04:53.806968 13667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b0428319a4bf3822912921671eaf3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35289 } }
I20260202 15:04:53.807650 13662 sys_catalog.cc:565] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:04:53.808529 13668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "370b0428319a4bf3822912921671eaf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b0428319a4bf3822912921671eaf3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35289 } } }
I20260202 15:04:53.808641 13668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [sys.catalog]: This master's current role is: LEADER
I20260202 15:04:53.808774 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:53.796051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.2.164.126:35289)
0202 15:04:53.796491 (+ 440us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:53.796501 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:53.796536 (+ 35us) server_negotiation.cc:408] Connection header received
0202 15:04:53.797173 (+ 637us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:53.797192 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:53.797492 (+ 300us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:53.797815 (+ 323us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:53.798758 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:53.799485 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:53.800163 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:53.800458 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:53.802849 (+ 2391us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:53.802870 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:53.802883 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:53.802921 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:53.805277 (+ 2356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:53.805769 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:53.805774 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:53.805799 (+ 25us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:53.805905 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:53.806160 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:53.806164 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:53.806166 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:53.806633 (+ 467us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:53.807508 (+ 875us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:53.807639 (+ 131us) server_negotiation.cc:300] Negotiation successful
0202 15:04:53.807909 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":152,"threads_started":1}
I20260202 15:04:53.808985 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370b0428319a4bf3822912921671eaf3. Latest consensus state: current_term: 1 leader_uuid: "370b0428319a4bf3822912921671eaf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b0428319a4bf3822912921671eaf3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35289 } } }
I20260202 15:04:53.809113 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3 [sys.catalog]: This master's current role is: LEADER
W20260202 15:04:53.812939 13682 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260202 15:04:53.813089 13682 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260202 15:04:53.813165 13672 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:04:53.814061 13672 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:04:53.819578 13672 catalog_manager.cc:1357] Generated new cluster ID: 38bb09c488fd4bf081959f1b93054a21
I20260202 15:04:53.819660 13672 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:04:53.863015 13672 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:04:53.864362 13672 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:04:53.871073 13672 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: Generated new TSK 0
I20260202 15:04:53.871953 13672 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:53.929417 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35289
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:54.038903 13690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:54.039145 13690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:54.039206 13690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:54.042531 13690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:54.042594 13690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:54.042718 13690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:04:54.047008 13690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:35289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:54.048156 13690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:54.049072 13690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:54.056274 13695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:54.056208 13698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:54.056208 13696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:54.056424 13690 server_base.cc:1060] running on GCE node
I20260202 15:04:54.056777 13690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:54.057404 13690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:54.058573 13690 hybrid_clock.cc:648] HybridClock initialized: now 1770044694058561 us; error 33 us; skew 500 ppm
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:54.061332 13690 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:04:54.062551 13690 webserver.cc:492] Webserver started at http://127.2.164.65:44321/ using document root <none> and password file <none>
I20260202 15:04:54.063184 13690 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:54.063267 13690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:54.063483 13690 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:54.065348 13690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "ee337ff9d02d4e7bb154103cdab92348"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "8c46a8877d96fcd243a057e70a409562"
server_key_iv: "4f869d1d41d7fb745cc235ad8a138262"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.065864 13690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "ee337ff9d02d4e7bb154103cdab92348"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "8c46a8877d96fcd243a057e70a409562"
server_key_iv: "4f869d1d41d7fb745cc235ad8a138262"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.069455 13690 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:54.071815 13705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.072935 13690 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:54.073114 13690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "ee337ff9d02d4e7bb154103cdab92348"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "8c46a8877d96fcd243a057e70a409562"
server_key_iv: "4f869d1d41d7fb745cc235ad8a138262"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.073232 13690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:54.084583 13690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:54.087661 13690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:54.087875 13690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:54.088467 13690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:54.089501 13690 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:54.089573 13690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.089645 13690 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:54.089699 13690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.099020 13690 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:44329
I20260202 15:04:54.099035 13818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:44329 every 8 connection(s)
I20260202 15:04:54.099980 13690 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:04:54.105638 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13690
I20260202 15:04:54.105804 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:04:54.106065 2705 external_mini_cluster.cc:1468] Setting key a66c82ad57bcd6f8698a7dcd206abf48
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[13578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:54.113556 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.101678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:57829 (local address 127.2.164.126:35289)
0202 15:04:54.101859 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.101863 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.102452 (+ 589us) server_negotiation.cc:408] Connection header received
0202 15:04:54.103224 (+ 772us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.103228 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.103276 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.103348 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:54.104962 (+ 1614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.105602 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.106373 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.106544 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.109344 (+ 2800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:54.109369 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:54.109375 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:54.109411 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:54.111286 (+ 1875us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.111875 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.111881 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.111884 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.111939 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.112217 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.112223 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.112227 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.112455 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:54.112584 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.113197 (+ 613us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.113352 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260202 15:04:54.114136 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.101909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35289 (local address 127.2.164.65:57829)
0202 15:04:54.102317 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.102351 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.103054 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.103498 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.103505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.103901 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:54.104826 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.104839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.105793 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.105796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.106266 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.106274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.106474 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.107100 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:54.107125 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:54.109139 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:54.111484 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:54.111493 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:54.111508 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:54.111767 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:54.112037 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:54.112039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:54.112041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:54.112123 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:54.112603 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:54.112609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:54.112849 (+ 240us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.113127 (+ 278us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.113359 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260202 15:04:54.115409 13819 heartbeater.cc:344] Connected to a master server at 127.2.164.126:35289
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:04:54.115698 13819 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:54.116242 13819 heartbeater.cc:507] Master 127.2.164.126:35289 requested a full tablet report, sending...
I20260202 15:04:54.117808 13626 ts_manager.cc:194] Registered new tserver with Master: ee337ff9d02d4e7bb154103cdab92348 (127.2.164.65:44329)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260202 15:04:54.119331 13626 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:57829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:54.159716 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35289
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:54.264580 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:54.264860 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:54.264950 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:54.268210 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:54.268312 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:54.268432 13826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:04:54.272819 13826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:35289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:54.273969 13826 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:54.274822 13826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:54.281462 13834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:54.281526 13831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:54.281462 13832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:54.281837 13826 server_base.cc:1060] running on GCE node
I20260202 15:04:54.282208 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:54.282791 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:54.283967 13826 hybrid_clock.cc:648] HybridClock initialized: now 1770044694283962 us; error 29 us; skew 500 ppm
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:54.286636 13826 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:04:54.287689 13826 webserver.cc:492] Webserver started at http://127.2.164.66:45791/ using document root <none> and password file <none>
I20260202 15:04:54.288196 13826 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:54.288241 13826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:54.288388 13826 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:54.290068 13826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "bc4e9ffcc9004021854309ce840059ee"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "bd975a029be357700b4519e876278b0d"
server_key_iv: "d2d66650ea8f70f6ff9c4d2e6b25d5ea"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.290493 13826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "bc4e9ffcc9004021854309ce840059ee"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "bd975a029be357700b4519e876278b0d"
server_key_iv: "d2d66650ea8f70f6ff9c4d2e6b25d5ea"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.293761 13826 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260202 15:04:54.295794 13841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.296764 13826 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:04:54.296851 13826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "bc4e9ffcc9004021854309ce840059ee"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "bd975a029be357700b4519e876278b0d"
server_key_iv: "d2d66650ea8f70f6ff9c4d2e6b25d5ea"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.296926 13826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:54.306514 13826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:54.309780 13826 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:54.309976 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:54.310659 13826 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:54.311559 13826 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:54.311605 13826 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.311673 13826 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:54.311712 13826 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.320591 13826 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:34931
I20260202 15:04:54.320767 13954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:34931 every 8 connection(s)
I20260202 15:04:54.321703 13826 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:04:54.325549 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13826
I20260202 15:04:54.325688 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:04:54.325968 2705 external_mini_cluster.cc:1468] Setting key 97bd7028b1c97d5a216f33c25c0da127
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[13578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:54.334770 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.323542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46223 (local address 127.2.164.126:35289)
0202 15:04:54.323682 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.323686 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.324320 (+ 634us) server_negotiation.cc:408] Connection header received
0202 15:04:54.325158 (+ 838us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.325162 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.325216 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.325319 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:54.327404 (+ 2085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.327948 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.328939 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.329149 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.331366 (+ 2217us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:54.331382 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:54.331384 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:54.331416 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:54.332835 (+ 1419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.333356 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.333360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.333362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.333413 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.333701 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.333705 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.333707 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.333865 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:54.333968 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.334472 (+ 504us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.334607 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260202 15:04:54.335422 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.323769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35289 (local address 127.2.164.66:46223)
0202 15:04:54.324167 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.324202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.324910 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.325521 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.325533 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.326091 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:54.327235 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.327253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.328178 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.328185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.328809 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.328822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.329076 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.329776 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:54.329800 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:54.331208 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:54.332972 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:54.332977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:54.332989 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:54.333256 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:54.333506 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:54.333511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:54.333516 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:54.333604 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:54.333980 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:54.333989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:54.334229 (+ 240us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.334424 (+ 195us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.334684 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260202 15:04:54.336567 13955 heartbeater.cc:344] Connected to a master server at 127.2.164.126:35289
I20260202 15:04:54.336849 13955 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:54.337486 13955 heartbeater.cc:507] Master 127.2.164.126:35289 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260202 15:04:54.338609 13626 ts_manager.cc:194] Registered new tserver with Master: bc4e9ffcc9004021854309ce840059ee (127.2.164.66:34931)
I20260202 15:04:54.339219 13626 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:46223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:04:54.376299 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35289
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:04:54.484323 13962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:04:54.484573 13962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:04:54.484684 13962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:04:54.488067 13962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:04:54.488140 13962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:04:54.488262 13962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:04:54.492550 13962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:35289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:04:54.493702 13962 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:04:54.494601 13962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:04:54.501346 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:04:54.501353 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:54.501519 13962 server_base.cc:1060] running on GCE node
W20260202 15:04:54.501667 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:04:54.502007 13962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:04:54.502487 13962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:04:54.503666 13962 hybrid_clock.cc:648] HybridClock initialized: now 1770044694503642 us; error 37 us; skew 500 ppm
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:04:54.506485 13962 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:04:54.507555 13962 webserver.cc:492] Webserver started at http://127.2.164.67:35611/ using document root <none> and password file <none>
I20260202 15:04:54.508155 13962 fs_manager.cc:362] Metadata directory not provided
I20260202 15:04:54.508213 13962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:04:54.508371 13962 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260202 15:04:54.510280 13962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "6183860dbfec48a8a999619b537cca10"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "ccbee493e4b28052447fbd8dc1305734"
server_key_iv: "404067eeb8b7a5a38c4b9a368f89d25e"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.510727 13962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "6183860dbfec48a8a999619b537cca10"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "ccbee493e4b28052447fbd8dc1305734"
server_key_iv: "404067eeb8b7a5a38c4b9a368f89d25e"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.514650 13962 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260202 15:04:54.517055 13977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.518091 13962 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260202 15:04:54.518188 13962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "6183860dbfec48a8a999619b537cca10"
format_stamp: "Formatted at 2026-02-02 15:04:54 on dist-test-slave-6zzp"
server_key: "ccbee493e4b28052447fbd8dc1305734"
server_key_iv: "404067eeb8b7a5a38c4b9a368f89d25e"
server_key_version: "encryptionkey@0"
I20260202 15:04:54.518270 13962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:04:54.533771 13962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:04:54.536494 13962 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:04:54.536646 13962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:04:54.537182 13962 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:04:54.538067 13962 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:04:54.538115 13962 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.538182 13962 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:04:54.538221 13962 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:04:54.549485 13962 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45087
I20260202 15:04:54.549515 14090 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:45087 every 8 connection(s)
I20260202 15:04:54.550428 13962 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
I20260202 15:04:54.552130 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 13962
I20260202 15:04:54.552248 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:04:54.552501 2705 external_mini_cluster.cc:1468] Setting key e694ceb9ce98aa786e5597a7eb1a7d1e
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[13578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:04:54.562726 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.552145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:43325 (local address 127.2.164.126:35289)
0202 15:04:54.552277 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.552281 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.553169 (+ 888us) server_negotiation.cc:408] Connection header received
0202 15:04:54.554158 (+ 989us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.554161 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.554218 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.554312 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:54.555957 (+ 1645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.556452 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.557147 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.557331 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.559556 (+ 2225us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:54.559578 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:54.559581 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:54.559605 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:54.560853 (+ 1248us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.561356 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.561361 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.561363 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.561414 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.561707 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.561711 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.561713 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.561868 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:54.561962 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.562445 (+ 483us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.562558 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260202 15:04:54.563300 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.552474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35289 (local address 127.2.164.67:43325)
0202 15:04:54.553000 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.553067 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.553968 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.554468 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.554480 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.554885 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:04:54.555818 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.555832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.556573 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.556579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.556979 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.557044 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.557262 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.557880 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:04:54.557904 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:04:54.559407 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:04:54.560987 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:54.560993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:54.561023 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:54.561256 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:54.561519 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:04:54.561521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:04:54.561523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:04:54.561604 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:04:54.561976 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:04:54.561984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:04:54.562208 (+ 224us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.562394 (+ 186us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.562654 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260202 15:04:54.564365 14091 heartbeater.cc:344] Connected to a master server at 127.2.164.126:35289
I20260202 15:04:54.564597 14091 heartbeater.cc:461] Registering TS with master...
I20260202 15:04:54.565233 14091 heartbeater.cc:507] Master 127.2.164.126:35289 requested a full tablet report, sending...
I20260202 15:04:54.566269 13625 ts_manager.cc:194] Registered new tserver with Master: 6183860dbfec48a8a999619b537cca10 (127.2.164.67:45087)
I20260202 15:04:54.566813 13625 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:43325
I20260202 15:04:54.578079 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260202 15:04:54.586620 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.579973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.2.164.126:35289)
0202 15:04:54.580123 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.580127 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.580236 (+ 109us) server_negotiation.cc:408] Connection header received
0202 15:04:54.580380 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.580384 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.580435 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.580516 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:04:54.581393 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.581886 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.582541 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.582753 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.583776 (+ 1023us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:04:54.583793 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:04:54.583796 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:04:54.583825 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:04:54.585135 (+ 1310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.585509 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.585513 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.585515 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.585578 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:04:54.585823 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:04:54.585827 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:04:54.585829 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:04:54.586002 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:04:54.586135 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.586339 (+ 204us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.586449 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260202 15:04:54.589253 13625 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:04:54.591674 13625 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:04:54.604720 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.600273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:34931 (local address 127.0.0.1:60814)
0202 15:04:54.600622 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.600666 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.600821 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.601662 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.601668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.601694 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:54.602030 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.602039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.603382 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.603387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.604197 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.604206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.604312 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.604375 (+ 63us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.604469 (+ 94us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.604569 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260202 15:04:54.605517 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.600400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.2.164.66:34931)
0202 15:04:54.601171 (+ 771us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.601178 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.601202 (+ 24us) server_negotiation.cc:408] Connection header received
0202 15:04:54.601286 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.601292 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.601491 (+ 199us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.601655 (+ 164us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:54.602183 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.603276 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.604351 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.604915 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.605246 (+ 331us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.605301 (+ 55us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.605374 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":148,"threads_started":1}
I20260202 15:04:54.605796 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.600476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45087 (local address 127.0.0.1:49312)
0202 15:04:54.601871 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.601885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.602294 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.602897 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.602900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.602918 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:54.603144 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.603150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.604463 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.604468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.605400 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.605410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.605519 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.605536 (+ 17us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.605590 (+ 54us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.605646 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"thread_start_us":93,"threads_started":1}
I20260202 15:04:54.606650 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.601051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.2.164.67:45087)
0202 15:04:54.601369 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.601375 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.602071 (+ 696us) server_negotiation.cc:408] Connection header received
0202 15:04:54.602452 (+ 381us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.602458 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.602622 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.602756 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:54.603273 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.604345 (+ 1072us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.605555 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.606231 (+ 676us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.606338 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.606402 (+ 64us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.606496 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":148,"threads_started":1}
I20260202 15:04:54.607013 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.601858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44329 (local address 127.0.0.1:42168)
0202 15:04:54.602075 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.602090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.602169 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.602681 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.602684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.602702 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:54.602934 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.602941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.604447 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.604452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.605526 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.605533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.606484 (+ 951us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.606496 (+ 12us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.606590 (+ 94us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.606651 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260202 15:04:54.607566 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.602032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.2.164.65:44329)
0202 15:04:54.602290 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.602297 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.602319 (+ 22us) server_negotiation.cc:408] Connection header received
0202 15:04:54.602402 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.602409 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.602543 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.602692 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:54.603044 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.604310 (+ 1266us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.605764 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.606338 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.606421 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.606642 (+ 221us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.606746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260202 15:04:54.608669 13889 tablet_service.cc:1505] Processing CreateTablet for tablet 485feccbf06b40a9be7804c2887733bb (DEFAULT_TABLE table=test-table [id=f671610865af4d65a09b1f97d4c253a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:54.609339 14025 tablet_service.cc:1505] Processing CreateTablet for tablet 485feccbf06b40a9be7804c2887733bb (DEFAULT_TABLE table=test-table [id=f671610865af4d65a09b1f97d4c253a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:54.609793 13889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485feccbf06b40a9be7804c2887733bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:54.610219 14025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485feccbf06b40a9be7804c2887733bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:54.610103 13753 tablet_service.cc:1505] Processing CreateTablet for tablet 485feccbf06b40a9be7804c2887733bb (DEFAULT_TABLE table=test-table [id=f671610865af4d65a09b1f97d4c253a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:04:54.610931 13753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485feccbf06b40a9be7804c2887733bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:04:54.615234 14108 tablet_bootstrap.cc:492] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Bootstrap starting.
I20260202 15:04:54.616737 14109 tablet_bootstrap.cc:492] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Bootstrap starting.
I20260202 15:04:54.617293 14110 tablet_bootstrap.cc:492] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348: Bootstrap starting.
I20260202 15:04:54.617820 14108 tablet_bootstrap.cc:654] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:54.618665 14108 log.cc:826] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:54.618947 14110 tablet_bootstrap.cc:654] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:54.618961 14109 tablet_bootstrap.cc:654] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Neither blocks nor log segments found. Creating new log.
I20260202 15:04:54.619601 14110 log.cc:826] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:54.619727 14109 log.cc:826] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Log is configured to *not* fsync() on all Append() calls
I20260202 15:04:54.620658 14108 tablet_bootstrap.cc:492] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: No bootstrap required, opened a new log
I20260202 15:04:54.620846 14108 ts_tablet_manager.cc:1403] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260202 15:04:54.621238 14110 tablet_bootstrap.cc:492] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348: No bootstrap required, opened a new log
I20260202 15:04:54.621412 14110 ts_tablet_manager.cc:1403] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:04:54.621718 14109 tablet_bootstrap.cc:492] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: No bootstrap required, opened a new log
I20260202 15:04:54.621892 14109 ts_tablet_manager.cc:1403] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260202 15:04:54.623915 14110 raft_consensus.cc:359] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.624109 14110 raft_consensus.cc:385] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:54.624162 14110 raft_consensus.cc:740] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee337ff9d02d4e7bb154103cdab92348, State: Initialized, Role: FOLLOWER
I20260202 15:04:54.624578 14110 consensus_queue.cc:260] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.624673 14108 raft_consensus.cc:359] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.624873 14108 raft_consensus.cc:385] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:54.624938 14108 raft_consensus.cc:740] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6183860dbfec48a8a999619b537cca10, State: Initialized, Role: FOLLOWER
I20260202 15:04:54.625344 13819 heartbeater.cc:499] Master 127.2.164.126:35289 was elected leader, sending a full tablet report...
I20260202 15:04:54.625356 14108 consensus_queue.cc:260] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.625419 14109 raft_consensus.cc:359] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.625546 14110 ts_tablet_manager.cc:1434] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260202 15:04:54.625577 14109 raft_consensus.cc:385] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:04:54.625639 14109 raft_consensus.cc:740] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4e9ffcc9004021854309ce840059ee, State: Initialized, Role: FOLLOWER
I20260202 15:04:54.626085 14091 heartbeater.cc:499] Master 127.2.164.126:35289 was elected leader, sending a full tablet report...
I20260202 15:04:54.626048 14109 consensus_queue.cc:260] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.626195 14108 ts_tablet_manager.cc:1434] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260202 15:04:54.626987 13955 heartbeater.cc:499] Master 127.2.164.126:35289 was elected leader, sending a full tablet report...
I20260202 15:04:54.628705 14109 ts_tablet_manager.cc:1434] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260202 15:04:54.739840 14115 raft_consensus.cc:493] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:04:54.740067 14115 raft_consensus.cc:515] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.741281 14115 leader_election.cc:290] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc4e9ffcc9004021854309ce840059ee (127.2.164.66:34931), ee337ff9d02d4e7bb154103cdab92348 (127.2.164.65:44329)
I20260202 15:04:54.744674 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.741586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:34931 (local address 127.2.164.67:36149)
0202 15:04:54.741742 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.741759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.741874 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.742204 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.742207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.742261 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:54.742548 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.742558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.743623 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.743628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.744204 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.744211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.744309 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.744356 (+ 47us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.744408 (+ 52us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.744464 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260202 15:04:54.744714 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.741586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44329 (local address 127.2.164.67:56889)
0202 15:04:54.742168 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.742183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:04:54.742278 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:04:54.742567 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:04:54.742571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:04:54.742588 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:04:54.742916 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.742924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.743647 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.743650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:04:54.744233 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:04:54.744240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.744380 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.744399 (+ 19us) client_negotiation.cc:769] Sending connection context
0202 15:04:54.744452 (+ 53us) client_negotiation.cc:240] Negotiation successful
0202 15:04:54.744517 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":191,"thread_start_us":160,"threads_started":1}
I20260202 15:04:54.745149 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.741702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:36149 (local address 127.2.164.66:34931)
0202 15:04:54.741870 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.741874 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.741895 (+ 21us) server_negotiation.cc:408] Connection header received
0202 15:04:54.742005 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.742010 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.742072 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.742162 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:54.742723 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.743476 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.744348 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.744841 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.744890 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.744944 (+ 54us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.745040 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260202 15:04:54.745183 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.741691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56889 (local address 127.2.164.65:44329)
0202 15:04:54.741828 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.741833 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.742281 (+ 448us) server_negotiation.cc:408] Connection header received
0202 15:04:54.742366 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.742369 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.742421 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.742511 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:04:54.743043 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.743514 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.744392 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.744891 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.744937 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.744993 (+ 56us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.745089 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260202 15:04:54.745729 13909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485feccbf06b40a9be7804c2887733bb" candidate_uuid: "6183860dbfec48a8a999619b537cca10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4e9ffcc9004021854309ce840059ee" is_pre_election: true
I20260202 15:04:54.745728 13773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485feccbf06b40a9be7804c2887733bb" candidate_uuid: "6183860dbfec48a8a999619b537cca10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee337ff9d02d4e7bb154103cdab92348" is_pre_election: true
I20260202 15:04:54.746009 13773 raft_consensus.cc:2468] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6183860dbfec48a8a999619b537cca10 in term 0.
I20260202 15:04:54.746009 13909 raft_consensus.cc:2468] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6183860dbfec48a8a999619b537cca10 in term 0.
I20260202 15:04:54.746486 13979 leader_election.cc:304] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6183860dbfec48a8a999619b537cca10, ee337ff9d02d4e7bb154103cdab92348; no voters:
I20260202 15:04:54.746758 14115 raft_consensus.cc:2804] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:04:54.746836 14115 raft_consensus.cc:493] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:04:54.746884 14115 raft_consensus.cc:3060] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:54.747869 14115 raft_consensus.cc:515] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.748288 14115 leader_election.cc:290] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [CANDIDATE]: Term 1 election: Requested vote from peers bc4e9ffcc9004021854309ce840059ee (127.2.164.66:34931), ee337ff9d02d4e7bb154103cdab92348 (127.2.164.65:44329)
I20260202 15:04:54.748687 13909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485feccbf06b40a9be7804c2887733bb" candidate_uuid: "6183860dbfec48a8a999619b537cca10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4e9ffcc9004021854309ce840059ee"
I20260202 15:04:54.748708 13773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "485feccbf06b40a9be7804c2887733bb" candidate_uuid: "6183860dbfec48a8a999619b537cca10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee337ff9d02d4e7bb154103cdab92348"
I20260202 15:04:54.748837 13909 raft_consensus.cc:3060] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:54.748837 13773 raft_consensus.cc:3060] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:04:54.750249 13909 raft_consensus.cc:2468] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6183860dbfec48a8a999619b537cca10 in term 1.
I20260202 15:04:54.750586 13773 raft_consensus.cc:2468] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6183860dbfec48a8a999619b537cca10 in term 1.
I20260202 15:04:54.750612 13981 leader_election.cc:304] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6183860dbfec48a8a999619b537cca10, bc4e9ffcc9004021854309ce840059ee; no voters:
I20260202 15:04:54.750850 14115 raft_consensus.cc:2804] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:04:54.751075 14115 raft_consensus.cc:697] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 1 LEADER]: Becoming Leader. State: Replica: 6183860dbfec48a8a999619b537cca10, State: Running, Role: LEADER
I20260202 15:04:54.751479 14115 consensus_queue.cc:237] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } }
I20260202 15:04:54.754827 13625 catalog_manager.cc:5654] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6183860dbfec48a8a999619b537cca10 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "6183860dbfec48a8a999619b537cca10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6183860dbfec48a8a999619b537cca10" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 } health_report { overall_health: UNKNOWN } } }
I20260202 15:04:54.800755 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.797495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.2.164.67:45087)
0202 15:04:54.797646 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0202 15:04:54.797650 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:04:54.797668 (+ 18us) server_negotiation.cc:408] Connection header received
0202 15:04:54.797782 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:04:54.797787 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:04:54.797848 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:04:54.797941 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:04:54.798380 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.798891 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:04:54.799553 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:04:54.799707 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:04:54.799802 (+ 95us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:04:54.800249 (+ 447us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:04:54.800348 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0202 15:04:54.800482 (+ 134us) server_negotiation.cc:300] Negotiation successful
0202 15:04:54.800573 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260202 15:04:54.801297 14092 tablet.cc:2378] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:04:54.806438 13773 raft_consensus.cc:1275] T 485feccbf06b40a9be7804c2887733bb P ee337ff9d02d4e7bb154103cdab92348 [term 1 FOLLOWER]: Refusing update from remote peer 6183860dbfec48a8a999619b537cca10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:54.806438 13909 raft_consensus.cc:1275] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 1 FOLLOWER]: Refusing update from remote peer 6183860dbfec48a8a999619b537cca10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:04:54.807125 14115 consensus_queue.cc:1048] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee337ff9d02d4e7bb154103cdab92348" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:54.807313 14118 consensus_queue.cc:1048] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4e9ffcc9004021854309ce840059ee" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:04:54.824038 13624 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:54.824334 13624 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:04:54.826831 13624 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 370b0428319a4bf3822912921671eaf3: Sending DeleteTablet for 3 replicas of tablet 485feccbf06b40a9be7804c2887733bb
I20260202 15:04:54.827759 13753 tablet_service.cc:1552] Processing DeleteTablet for tablet 485feccbf06b40a9be7804c2887733bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:54 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:42168
I20260202 15:04:54.827800 13889 tablet_service.cc:1552] Processing DeleteTablet for tablet 485feccbf06b40a9be7804c2887733bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:54 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60814
I20260202 15:04:54.827869 14025 tablet_service.cc:1552] Processing DeleteTablet for tablet 485feccbf06b40a9be7804c2887733bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:04:54 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:49312
I20260202 15:04:54.828482 14134 tablet_replica.cc:333] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: stopping tablet replica
I20260202 15:04:54.828766 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13690
I20260202 15:04:54.829878 14133 tablet_replica.cc:333] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: stopping tablet replica
I20260202 15:04:54.830188 14133 raft_consensus.cc:2243] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:04:54.830564 14133 raft_consensus.cc:2272] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:54.830926 14134 raft_consensus.cc:2243] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:04:54.831346 14134 raft_consensus.cc:2272] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:04:54.832279 14133 ts_tablet_manager.cc:1916] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:54.832800 14134 ts_tablet_manager.cc:1916] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260202 15:04:54.835026 14133 ts_tablet_manager.cc:1929] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:54.835129 14133 log.cc:1199] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-1/wal/wals/485feccbf06b40a9be7804c2887733bb
I20260202 15:04:54.835312 14134 ts_tablet_manager.cc:1929] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:04:54.835409 14134 log.cc:1199] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/485feccbf06b40a9be7804c2887733bb
I20260202 15:04:54.835476 14133 ts_tablet_manager.cc:1950] T 485feccbf06b40a9be7804c2887733bb P bc4e9ffcc9004021854309ce840059ee: Deleting consensus metadata
I20260202 15:04:54.835763 14134 ts_tablet_manager.cc:1950] T 485feccbf06b40a9be7804c2887733bb P 6183860dbfec48a8a999619b537cca10: Deleting consensus metadata
I20260202 15:04:54.836860 13613 catalog_manager.cc:4985] TS bc4e9ffcc9004021854309ce840059ee (127.2.164.66:34931): tablet 485feccbf06b40a9be7804c2887733bb (table test-table [id=f671610865af4d65a09b1f97d4c253a0]) successfully deleted
I20260202 15:04:54.836860 13611 catalog_manager.cc:4985] TS 6183860dbfec48a8a999619b537cca10 (127.2.164.67:45087): tablet 485feccbf06b40a9be7804c2887733bb (table test-table [id=f671610865af4d65a09b1f97d4c253a0]) successfully deleted
W20260202 15:04:54.837999 13611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:44329 (error 108)
I20260202 15:04:54.838078 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13826
I20260202 15:04:54.841962 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:04:54.841619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44329 (local address 127.0.0.1:42176)
0202 15:04:54.841785 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0202 15:04:54.841862 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260202 15:04:54.842229 13611 catalog_manager.cc:4712] TS ee337ff9d02d4e7bb154103cdab92348 (127.2.164.65:44329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 485feccbf06b40a9be7804c2887733bb: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44329: connect: Connection refused (error 111)
I20260202 15:04:54.844767 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13962
I20260202 15:04:54.850513 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 13594
2026-02-02T15:04:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3407 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[14138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[14138](info): set up 2 sockets
Feb 02 15:04:54 dist-test-slave-6zzp krb5kdc[14138](info): commencing operation
krb5kdc: starting...
W20260202 15:04:56.845860 2705 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:04:56 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:04:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:04:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
I20260202 15:04:57.011400 2705 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRqWm_m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/postgres -l logfile start
2026-02-02 15:04:59.649 UTC [14169] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-02 15:04:59.649 UTC [14169] LOG: listening on IPv4 address "127.2.164.84", port 44153
2026-02-02 15:04:59.652 UTC [14169] LOG: listening on Unix socket "/tmp/.s.PGSQL.44153"
2026-02-02 15:04:59.660 UTC [14174] LOG: database system was shut down at 2026-02-02 15:04:58 UTC
2026-02-02 15:04:59.665 UTC [14169] LOG: database system is ready to accept connections
I20260202 15:05:01.573624 2705 mini_postgres.cc:96] Postgres bound to 44153
2026-02-02 15:05:01.579 UTC [14180] FATAL: database "slave" does not exist
127.2.164.84:44153 - accepting connections
I20260202 15:05:01.580525 2705 mini_ranger.cc:161] Starting Ranger...
I20260202 15:05:01.598240 2705 mini_ranger.cc:84] Created miniranger Postgres user
I20260202 15:05:01.669967 2705 mini_ranger.cc:87] Created ranger Postgres database
I20260202 15:05:01.670095 2705 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-admin
2026-02-02 15:05:01,960 [I] DB FLAVOR :POSTGRES
2026-02-02 15:05:01,961 [I] --------- Verifying Ranger DB connection ---------
2026-02-02 15:05:01,961 [I] Checking connection..
2026-02-02 15:05:01,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-02 15:05:02,280 [I] Checking connection passed.
2026-02-02 15:05:02,280 [I] --------- Verifying version history table ---------
2026-02-02 15:05:02,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-02 15:05:02,577 [I] Table x_db_version_h does not exist in database ranger
2026-02-02 15:05:02,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-02 15:05:02,904 [I] Table x_db_version_h does not exist in database ranger
2026-02-02 15:05:02,904 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-02 15:05:02,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-02 15:05:03.200 UTC [14247] WARNING: there is no transaction in progress
2026-02-02 15:05:03,207 [I] create_dbversion_catalog.sql file imported successfully
2026-02-02 15:05:03,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-02 15:05:03,504 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-02 15:05:03,505 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-02 15:05:03,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-02 15:05:03,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-02 15:05:04,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6zzp.c.gcp-upstream.internal','N') ;"
2026-02-02 15:05:04,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-02 15:05:04,732 [I] Table x_portal_user does not exist in database ranger
2026-02-02 15:05:04,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-02 15:05:05,043 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-02 15:05:05,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-02 15:05:05,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-02 15:05:05,646 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-02 15:05:05,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-02 15:05:06.480 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.495 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.510 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.525 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.538 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.552 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.599 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.605 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.616 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.625 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.635 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.646 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.654 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.660 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.663 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:06.667 UTC [14364] WARNING: there is no transaction in progress
2026-02-02 15:05:07,096 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-02 15:05:07,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-02 15:05:07,378 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-02 15:05:07,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-02 15:05:07,679 [I] Table x_portal_user already exists in database 'ranger'
2026-02-02 15:05:07,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-02 15:05:07,973 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-02 15:05:07,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-02 15:05:08,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-02 15:05:08,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6zzp.c.gcp-upstream.internal';"
2026-02-02 15:05:08,849 [I] CORE_DB_SCHEMA import status has been updated
2026-02-02 15:05:08,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-02 15:05:09,159 [I] DB_PATCHES have already been applied
I20260202 15:05:09.165110 2705 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRqWm_m/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260202 15:05:09.165169 2705 mini_ranger.cc:193] Using host: 127.2.164.84
I20260202 15:05:09.168042 2705 mini_ranger.cc:227] Ranger admin URL: http://127.2.164.84:39861
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp].
Feb 02, 2026 3:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260202 15:05:11.384802 2705 mini_ranger.cc:160] Time spent starting Ranger: real 9.804s user 0.006s sys 0.004s
Feb 02, 2026 3:05:30 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 02, 2026 3:05:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
Feb 02, 2026 3:05:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 02, 2026 3:05:30 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 02, 2026 3:05:31 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260202 15:05:31.958408 2705 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.164.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 02 15:05:32 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044732, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260202 15:05:32.057248 2705 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260202 15:05:32.069885 2705 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260202 15:05:32.191620 2705 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260202 15:05:32.191726 2705 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms
I20260202 15:05:32.191771 2705 mini_ranger_kms.cc:227] Using postgres at 127.2.164.84:44153
2026-02-02 15:05:32,395 [I] DB FLAVOR :POSTGRES
2026-02-02 15:05:32,395 [I] --------- Verifying Ranger DB connection ---------
2026-02-02 15:05:32,396 [I] Checking connection
2026-02-02 15:05:32,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-02 15:05:32,735 [I] connection success
2026-02-02 15:05:32,735 [I] --------- Verifying Ranger DB tables ---------
2026-02-02 15:05:32,735 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-02 15:05:32,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-02 15:05:33,048 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-02 15:05:33,048 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-02 15:05:33,048 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-02 15:05:33,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:44153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-02 15:05:33,360 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260202 15:05:33.857720 2705 mini_ranger_kms.cc:310] Created kms service
I20260202 15:05:34.051879 2705 mini_ranger_kms.cc:326] Created kudu user
I20260202 15:05:34.094424 2705 mini_ranger_kms.cc:343] Created rangerkms user
Feb 02 15:05:34 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044732, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:05:34.463079 2705 mini_ranger_kms.cc:384] Added ranger policy
I20260202 15:05:34.463243 2705 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260202 15:05:34.463266 2705 mini_ranger_kms.cc:240] Using host: 127.2.164.84
I20260202 15:05:34.466912 2705 mini_ranger_kms.cc:276] Ranger KMS PID: 14556
I20260202 15:05:34.467015 2705 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.164.84:35369
Feb 02, 2026 3:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 02, 2026 3:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 02, 2026 3:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 02, 2026 3:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 02, 2026 3:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-kms/ews/webapp].
Feb 02, 2026 3:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260202 15:05:36.693945 2705 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.637s user 0.016s sys 0.168s
I20260202 15:05:36.694094 2705 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260202 15:05:36.694187 2705 mini_ranger_kms.cc:401] 127.2.164.84:35369/kms/v1/keys
Feb 02 15:05:36 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1770044736, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 02 15:05:37 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044736, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Feb 02 15:05:37 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044736, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Feb 02 15:05:39 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:05:39.911753 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:46089
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:46089
--ranger_config_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:35369/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:05:40.028024 14609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:05:40.028264 14609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:05:40.028318 14609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:05:40.031553 14609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:05:40.031625 14609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:05:40.031649 14609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:05:40.031667 14609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:05:40.031684 14609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:05:40.036195 14609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.164.84:35369/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:46089
--ranger_config_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:05:40.037331 14609 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:05:40.038137 14609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:05:40.044335 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:40.044312 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:40.044304 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:05:40.044586 14609 server_base.cc:1060] running on GCE node
I20260202 15:05:40.045186 14609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:05:40.046164 14609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:05:40.047372 14609 hybrid_clock.cc:648] HybridClock initialized: now 1770044740047349 us; error 40 us; skew 500 ppm
Feb 02 15:05:40 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044740, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:05:40.049997 14609 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:05:40.051024 14609 webserver.cc:492] Webserver started at http://127.2.164.126:43895/ using document root <none> and password file <none>
I20260202 15:05:40.051569 14609 fs_manager.cc:362] Metadata directory not provided
I20260202 15:05:40.051643 14609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:05:40.051856 14609 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:05:40 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044740, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:05:40.106215 14609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "66b44d1bb77342579b44a9b25a537a88"
format_stamp: "Formatted at 2026-02-02 15:05:40 on dist-test-slave-6zzp"
server_key: "69357f5e108b534b985e9ddaef7602ae"
server_key_iv: "d18d48e03317488d640b862021e5746b"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:40.106878 14609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "66b44d1bb77342579b44a9b25a537a88"
format_stamp: "Formatted at 2026-02-02 15:05:40 on dist-test-slave-6zzp"
server_key: "69357f5e108b534b985e9ddaef7602ae"
server_key_iv: "d18d48e03317488d640b862021e5746b"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:40.111682 14609 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260202 15:05:40.128820 14624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:40.130214 14609 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260202 15:05:40.130366 14609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "66b44d1bb77342579b44a9b25a537a88"
format_stamp: "Formatted at 2026-02-02 15:05:40 on dist-test-slave-6zzp"
server_key: "69357f5e108b534b985e9ddaef7602ae"
server_key_iv: "d18d48e03317488d640b862021e5746b"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:40.130499 14609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:05:40.151067 14609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:05:40.158078 14609 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:05:40.158317 14609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:05:40.168560 14676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:46089 every 8 connection(s)
I20260202 15:05:40.168601 14609 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:46089
I20260202 15:05:40.169669 14609 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:05:40.169893 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 14609
I20260202 15:05:40.170039 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:05:40.170322 2705 external_mini_cluster.cc:1468] Setting key 431f55743aa17961b274b7f0c55c2884
I20260202 15:05:40.172735 14677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Feb 02 15:05:40 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:05:40.179247 14677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88: Bootstrap starting.
I20260202 15:05:40.182039 14677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88: Neither blocks nor log segments found. Creating new log.
I20260202 15:05:40.182897 14677 log.cc:826] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88: Log is configured to *not* fsync() on all Append() calls
I20260202 15:05:40.183652 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:40.171587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.2.164.126:46089)
0202 15:05:40.171990 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:40.171999 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:40.172032 (+ 33us) server_negotiation.cc:408] Connection header received
0202 15:05:40.172653 (+ 621us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:40.172670 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:40.172988 (+ 318us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:40.173359 (+ 371us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:05:40.174299 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:40.175148 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:40.175907 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:40.176201 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:40.178861 (+ 2660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:05:40.178920 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:05:40.178933 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:05:40.178975 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:05:40.181073 (+ 2098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:40.181566 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:40.181573 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:40.181579 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:40.181659 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:40.181899 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:40.181901 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:40.181902 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:40.182279 (+ 377us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:05:40.182439 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:40.182704 (+ 265us) server_negotiation.cc:300] Negotiation successful
0202 15:05:40.182940 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260202 15:05:40.185074 14677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88: No bootstrap required, opened a new log
I20260202 15:05:40.187837 14677 raft_consensus.cc:359] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b44d1bb77342579b44a9b25a537a88" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46089 } }
I20260202 15:05:40.188035 14677 raft_consensus.cc:385] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:05:40.188125 14677 raft_consensus.cc:740] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b44d1bb77342579b44a9b25a537a88, State: Initialized, Role: FOLLOWER
I20260202 15:05:40.188565 14677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b44d1bb77342579b44a9b25a537a88" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46089 } }
I20260202 15:05:40.188705 14677 raft_consensus.cc:399] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:05:40.188782 14677 raft_consensus.cc:493] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:05:40.188892 14677 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:05:40.189924 14677 raft_consensus.cc:515] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b44d1bb77342579b44a9b25a537a88" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46089 } }
I20260202 15:05:40.190246 14677 leader_election.cc:304] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b44d1bb77342579b44a9b25a537a88; no voters:
I20260202 15:05:40.190546 14677 leader_election.cc:290] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:05:40.190642 14682 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:05:40.190899 14682 raft_consensus.cc:697] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [term 1 LEADER]: Becoming Leader. State: Replica: 66b44d1bb77342579b44a9b25a537a88, State: Running, Role: LEADER
I20260202 15:05:40.191198 14682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b44d1bb77342579b44a9b25a537a88" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46089 } }
I20260202 15:05:40.191684 14677 sys_catalog.cc:565] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:05:40.192835 14684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b44d1bb77342579b44a9b25a537a88. Latest consensus state: current_term: 1 leader_uuid: "66b44d1bb77342579b44a9b25a537a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b44d1bb77342579b44a9b25a537a88" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46089 } } }
I20260202 15:05:40.192966 14684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [sys.catalog]: This master's current role is: LEADER
I20260202 15:05:40.193367 14691 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:05:40.193518 14683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b44d1bb77342579b44a9b25a537a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b44d1bb77342579b44a9b25a537a88" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46089 } } }
I20260202 15:05:40.193598 14683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88 [sys.catalog]: This master's current role is: LEADER
I20260202 15:05:40.196466 14691 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:05:40.196893 14677 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260202 15:05:40.206804 14691 catalog_manager.cc:1357] Generated new cluster ID: bf6b5ba3056e488798ec4dd21d88b338
I20260202 15:05:40.206929 14691 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:05:40.216483 14691 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:05:40.217563 14691 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:05:40.224926 14691 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88: Generated new TSK 0
I20260202 15:05:40.225795 14691 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:05:40.996889 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46089
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:05:41.136663 14723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:05:41.136907 14723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:05:41.137008 14723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:05:41.140434 14723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:05:41.140501 14723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:05:41.140623 14723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:05:41.145203 14723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.164.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:05:41.146292 14723 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:05:41.147228 14723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:05:41.153913 14733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:41.153956 14731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:05:41.154202 14723 server_base.cc:1060] running on GCE node
W20260202 15:05:41.154942 14730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:05:41.155359 14723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:05:41.155939 14723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:05:41.157076 14723 hybrid_clock.cc:648] HybridClock initialized: now 1770044741157057 us; error 34 us; skew 500 ppm
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:05:41.159614 14723 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:05:41.160686 14723 webserver.cc:492] Webserver started at http://127.2.164.65:39417/ using document root <none> and password file <none>
I20260202 15:05:41.161415 14723 fs_manager.cc:362] Metadata directory not provided
I20260202 15:05:41.161484 14723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:05:41.161720 14723 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:05:41.187628 14723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "3e16181a211a4461905e6d8a841afd94"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "06062ef6388cb7f492b5448c4306c765"
server_key_iv: "683dc10520ef4057439928ba51d862b3"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.188158 14723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "3e16181a211a4461905e6d8a841afd94"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "06062ef6388cb7f492b5448c4306c765"
server_key_iv: "683dc10520ef4057439928ba51d862b3"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.193152 14723 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260202 15:05:41.220551 14740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.222117 14723 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260202 15:05:41.222263 14723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "3e16181a211a4461905e6d8a841afd94"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "06062ef6388cb7f492b5448c4306c765"
server_key_iv: "683dc10520ef4057439928ba51d862b3"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.222410 14723 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:05:41.242569 14723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:05:41.246145 14723 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:05:41.246359 14723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:05:41.247052 14723 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:05:41.248153 14723 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:05:41.248215 14723 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.248287 14723 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:05:41.248315 14723 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.263044 14723 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:35983
I20260202 15:05:41.264303 14723 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:05:41.268606 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 14723
I20260202 15:05:41.268757 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:05:41.269050 2705 external_mini_cluster.cc:1468] Setting key 2c2c04dc12a69ddeb89f6ea6692ced4f
I20260202 15:05:41.266309 14855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:35983 every 8 connection(s)
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:05:41.287523 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:41.268380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46089 (local address 127.2.164.65:51221)
0202 15:05:41.270176 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:41.270221 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:41.271242 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:41.273208 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:41.273220 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:41.273613 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:05:41.274407 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:41.274424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.275620 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:41.275624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:41.276258 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:41.276269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.276504 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:41.277618 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:05:41.277655 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:05:41.279473 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:05:41.284129 (+ 4656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:05:41.284137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:05:41.284154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:05:41.284576 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:05:41.284911 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:05:41.284915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:05:41.284917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:05:41.285112 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:05:41.285622 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:05:41.285631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:05:41.285969 (+ 338us) client_negotiation.cc:769] Sending connection context
0202 15:05:41.286222 (+ 253us) client_negotiation.cc:240] Negotiation successful
0202 15:05:41.286583 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":147,"threads_started":1}
I20260202 15:05:41.288443 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:41.269563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:51221 (local address 127.2.164.126:46089)
0202 15:05:41.272628 (+ 3065us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:41.272633 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:41.272649 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:05:41.272696 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:41.272700 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:41.272755 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:41.272850 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:05:41.274595 (+ 1745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.275463 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:41.277037 (+ 1574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.277244 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:41.279661 (+ 2417us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:05:41.279686 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:05:41.279689 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:05:41.279720 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:05:41.282206 (+ 2486us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:41.284726 (+ 2520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:41.284731 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:41.284733 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:41.284800 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:41.285231 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:41.285234 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:41.285236 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:41.285458 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:05:41.285559 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:41.288101 (+ 2542us) server_negotiation.cc:300] Negotiation successful
0202 15:05:41.288253 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2966,"thread_start_us":75,"threads_started":1}
I20260202 15:05:41.294726 14856 heartbeater.cc:344] Connected to a master server at 127.2.164.126:46089
I20260202 15:05:41.294997 14856 heartbeater.cc:461] Registering TS with master...
I20260202 15:05:41.295655 14856 heartbeater.cc:507] Master 127.2.164.126:46089 requested a full tablet report, sending...
I20260202 15:05:41.297456 14641 ts_manager.cc:194] Registered new tserver with Master: 3e16181a211a4461905e6d8a841afd94 (127.2.164.65:35983)
I20260202 15:05:41.299089 14641 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:51221
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:05:41.375267 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46089
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:05:41.509907 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:05:41.510213 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:05:41.510329 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:05:41.513978 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:05:41.514098 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:05:41.514240 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:05:41.519565 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.164.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:05:41.520953 14864 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:05:41.522042 14864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:05:41.529269 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:41.529556 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:41.529595 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:05:41.530681 14864 server_base.cc:1060] running on GCE node
I20260202 15:05:41.531082 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:05:41.531720 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:05:41.532927 14864 hybrid_clock.cc:648] HybridClock initialized: now 1770044741532903 us; error 36 us; skew 500 ppm
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:05:41.536206 14864 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:05:41.537572 14864 webserver.cc:492] Webserver started at http://127.2.164.66:46053/ using document root <none> and password file <none>
I20260202 15:05:41.538163 14864 fs_manager.cc:362] Metadata directory not provided
I20260202 15:05:41.538237 14864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:05:41.538445 14864 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:05:41.552032 14864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "7a4d60df67834be1a9d7a53387d93537"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "45d9bfba98ecbfb6035b71830daff46f"
server_key_iv: "11d4eba5824b4258d886dbb6ab56f24e"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.552716 14864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "7a4d60df67834be1a9d7a53387d93537"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "45d9bfba98ecbfb6035b71830daff46f"
server_key_iv: "11d4eba5824b4258d886dbb6ab56f24e"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.556653 14864 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260202 15:05:41.583259 14880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.584874 14864 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260202 15:05:41.585119 14864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "7a4d60df67834be1a9d7a53387d93537"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "45d9bfba98ecbfb6035b71830daff46f"
server_key_iv: "11d4eba5824b4258d886dbb6ab56f24e"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.585343 14864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:05:41.613099 14864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:05:41.616071 14864 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:05:41.616290 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:05:41.616856 14864 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:05:41.617816 14864 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:05:41.617893 14864 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.617960 14864 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:05:41.617995 14864 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.631057 14864 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:41945
I20260202 15:05:41.632275 14864 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:05:41.634475 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 14864
I20260202 15:05:41.634595 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:05:41.634871 2705 external_mini_cluster.cc:1468] Setting key 6ff39590b2c6959c29715ba92785de45
I20260202 15:05:41.636440 14993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:41945 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:05:41.667420 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:41.639197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46089 (local address 127.2.164.66:57061)
0202 15:05:41.646256 (+ 7059us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:41.646318 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:41.647490 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:41.648354 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:41.648378 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:41.648790 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:05:41.649734 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:41.649752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.650878 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:41.650881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:41.651477 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:41.651487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.656686 (+ 5199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:41.657388 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:05:41.657430 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:05:41.662040 (+ 4610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:05:41.664298 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:05:41.664305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:05:41.664320 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:05:41.664689 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:05:41.664978 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:05:41.664981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:05:41.664984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:05:41.665157 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:05:41.665642 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:05:41.665650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:05:41.665977 (+ 327us) client_negotiation.cc:769] Sending connection context
0202 15:05:41.666223 (+ 246us) client_negotiation.cc:240] Negotiation successful
0202 15:05:41.666524 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6754,"thread_start_us":123,"threads_started":1}
I20260202 15:05:41.668439 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:41.640247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:57061 (local address 127.2.164.126:46089)
0202 15:05:41.640402 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:41.640407 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:41.646459 (+ 6052us) server_negotiation.cc:408] Connection header received
0202 15:05:41.647769 (+ 1310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:41.647774 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:41.647837 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:41.647898 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:05:41.649925 (+ 2027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.650714 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:41.656167 (+ 5453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.656402 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:41.662254 (+ 5852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:05:41.662282 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:05:41.662288 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:05:41.662323 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:05:41.664140 (+ 1817us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:41.664816 (+ 676us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:41.664820 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:41.664823 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:41.664876 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:41.665270 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:41.665274 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:41.665276 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:41.665486 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:05:41.665582 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:41.668108 (+ 2526us) server_negotiation.cc:300] Negotiation successful
0202 15:05:41.668244 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260202 15:05:41.669497 14994 heartbeater.cc:344] Connected to a master server at 127.2.164.126:46089
I20260202 15:05:41.669783 14994 heartbeater.cc:461] Registering TS with master...
I20260202 15:05:41.670449 14994 heartbeater.cc:507] Master 127.2.164.126:46089 requested a full tablet report, sending...
I20260202 15:05:41.671554 14641 ts_manager.cc:194] Registered new tserver with Master: 7a4d60df67834be1a9d7a53387d93537 (127.2.164.66:41945)
I20260202 15:05:41.672334 14641 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:57061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:05:41.723757 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46089
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:44717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
W20260202 15:05:41.870056 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:05:41.870314 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:05:41.870414 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:05:41.874290 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:05:41.874382 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:05:41.874524 15001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260202 15:05:41.879024 15001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.164.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:05:41.880105 15001 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:05:41.881096 15001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:05:41.887939 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:41.888257 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:05:41.888262 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:05:41.888574 15001 server_base.cc:1060] running on GCE node
I20260202 15:05:41.888942 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:05:41.889581 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:05:41.890755 15001 hybrid_clock.cc:648] HybridClock initialized: now 1770044741890744 us; error 29 us; skew 500 ppm
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:05:41.894325 15001 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260202 15:05:41.896106 15001 webserver.cc:492] Webserver started at http://127.2.164.67:44687/ using document root <none> and password file <none>
I20260202 15:05:41.896745 15001 fs_manager.cc:362] Metadata directory not provided
I20260202 15:05:41.896799 15001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:05:41.897051 15001 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:05:41.911936 15001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data/instance:
uuid: "8cc8e4c7526347b6b6a669484d72c987"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "aa1f1f99ebb4be6cd2e5913fb3fb5bcb"
server_key_iv: "129e85dc77f6cbd43c8147c1092a261b"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.913209 15001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "8cc8e4c7526347b6b6a669484d72c987"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "aa1f1f99ebb4be6cd2e5913fb3fb5bcb"
server_key_iv: "129e85dc77f6cbd43c8147c1092a261b"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.918275 15001 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260202 15:05:41.935151 15016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.936702 15001 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260202 15:05:41.936858 15001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal
uuid: "8cc8e4c7526347b6b6a669484d72c987"
format_stamp: "Formatted at 2026-02-02 15:05:41 on dist-test-slave-6zzp"
server_key: "aa1f1f99ebb4be6cd2e5913fb3fb5bcb"
server_key_iv: "129e85dc77f6cbd43c8147c1092a261b"
server_key_version: "kuduclusterkey@0"
I20260202 15:05:41.937021 15001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:05:41.965405 15001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:05:41.968956 15001 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:05:41.969318 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:05:41.970031 15001 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:05:41.971099 15001 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:05:41.971169 15001 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:05:41.971227 15001 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:05:41.971318 15001 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260202 15:05:41.983500 15001 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43181
I20260202 15:05:41.983801 15129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.67:43181 every 8 connection(s)
I20260202 15:05:41.985350 15001 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/data/info.pb
Feb 02 15:05:41 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:05:41.994751 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 15001
I20260202 15:05:41.994884 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal/instance
I20260202 15:05:41.995137 2705 external_mini_cluster.cc:1468] Setting key 803535b3c19e9446f8cfbb1599d171e1
I20260202 15:05:41.998447 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:41.987285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:57793 (local address 127.2.164.126:46089)
0202 15:05:41.987430 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:41.987435 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:41.988224 (+ 789us) server_negotiation.cc:408] Connection header received
0202 15:05:41.989216 (+ 992us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:41.989219 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:41.989305 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:41.989401 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:05:41.990549 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.991124 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:41.991872 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.992054 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:41.994341 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:05:41.994364 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:05:41.994368 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:05:41.994398 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:05:41.996084 (+ 1686us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:41.996703 (+ 619us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:41.996707 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:41.996709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:41.996761 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:41.997141 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:41.997145 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:41.997147 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:41.997356 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:05:41.997435 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:41.998152 (+ 717us) server_negotiation.cc:300] Negotiation successful
0202 15:05:41.998277 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260202 15:05:41.999773 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:41.987631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46089 (local address 127.2.164.67:57793)
0202 15:05:41.988080 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:41.988123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:41.988952 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:41.989577 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:41.989586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:41.989850 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:05:41.990397 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:41.990409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.991243 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:41.991247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:41.991691 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:41.991699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:41.991992 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:41.992637 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:05:41.992661 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:05:41.994141 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:05:41.996225 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:05:41.996233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:05:41.996249 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:05:41.996584 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:05:41.996873 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:05:41.996876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:05:41.996879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:05:41.997023 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:05:41.997524 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:05:41.997531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:05:41.997864 (+ 333us) client_negotiation.cc:769] Sending connection context
0202 15:05:41.998609 (+ 745us) client_negotiation.cc:240] Negotiation successful
0202 15:05:41.998887 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
I20260202 15:05:42.000862 15130 heartbeater.cc:344] Connected to a master server at 127.2.164.126:46089
I20260202 15:05:42.001140 15130 heartbeater.cc:461] Registering TS with master...
I20260202 15:05:42.001681 15130 heartbeater.cc:507] Master 127.2.164.126:46089 requested a full tablet report, sending...
I20260202 15:05:42.002662 14641 ts_manager.cc:194] Registered new tserver with Master: 8cc8e4c7526347b6b6a669484d72c987 (127.2.164.67:43181)
I20260202 15:05:42.003223 14641 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:57793
I20260202 15:05:42.010913 2705 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260202 15:05:42.025954 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.017147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.2.164.126:46089)
0202 15:05:42.017280 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.017284 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.017300 (+ 16us) server_negotiation.cc:408] Connection header received
0202 15:05:42.017354 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.017358 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.017416 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.017510 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:05:42.018707 (+ 1197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.019497 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.020428 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.020621 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.022341 (+ 1720us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:05:42.022358 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:05:42.022362 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:05:42.022389 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:05:42.024144 (+ 1755us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:42.024642 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:42.024646 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:42.024649 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:42.024696 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:05:42.024968 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:05:42.024972 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:05:42.024974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:05:42.025175 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:05:42.025282 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.025663 (+ 381us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.025782 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260202 15:05:42.029346 14641 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260202 15:05:42.032631 14641 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260202 15:05:42.052368 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.044637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.2.164.67:43181)
0202 15:05:42.047800 (+ 3163us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.047806 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.047825 (+ 19us) server_negotiation.cc:408] Connection header received
0202 15:05:42.047891 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.047895 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.048022 (+ 127us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.048155 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:05:42.048715 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.049916 (+ 1201us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.051418 (+ 1502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.051930 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.052050 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.052116 (+ 66us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.052202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3055,"thread_start_us":117,"threads_started":1}
I20260202 15:05:42.052875 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.043838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43181 (local address 127.0.0.1:44860)
0202 15:05:42.045506 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.045563 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:42.045722 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:42.048292 (+ 2570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:42.048295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:42.048358 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:05:42.048580 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.048587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.050053 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.050057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:42.050961 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.050971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.051078 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.051133 (+ 55us) client_negotiation.cc:769] Sending connection context
0202 15:05:42.051218 (+ 85us) client_negotiation.cc:240] Negotiation successful
0202 15:05:42.052743 (+ 1525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"spinlock_wait_cycles":70656,"thread_start_us":124,"threads_started":1}
I20260202 15:05:42.056789 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.045122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35983 (local address 127.0.0.1:46914)
0202 15:05:42.047461 (+ 2339us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.047476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:42.047555 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:42.048292 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:42.048295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:42.048325 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:05:42.048580 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.048587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.055486 (+ 6899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.055490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:42.056359 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.056370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.056503 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.056524 (+ 21us) client_negotiation.cc:769] Sending connection context
0202 15:05:42.056576 (+ 52us) client_negotiation.cc:240] Negotiation successful
0202 15:05:42.056660 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"spinlock_wait_cycles":1280,"thread_start_us":95,"threads_started":1}
I20260202 15:05:42.056469 15064 tablet_service.cc:1505] Processing CreateTablet for tablet 51050225a0b44a0bacd41ac8e70f02cd (DEFAULT_TABLE table=test-table [id=ddccb485a59d400ea13055eb76f0124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:05:42.057507 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.045755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.2.164.65:35983)
0202 15:05:42.046296 (+ 541us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.046303 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.047768 (+ 1465us) server_negotiation.cc:408] Connection header received
0202 15:05:42.047857 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.047863 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.048022 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.048155 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:05:42.048727 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.049928 (+ 1201us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.056501 (+ 6573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.057087 (+ 586us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.057205 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.057272 (+ 67us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.057372 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":102,"threads_started":1}
I20260202 15:05:42.057561 15064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51050225a0b44a0bacd41ac8e70f02cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:05:42.060432 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.044018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41945 (local address 127.0.0.1:52732)
0202 15:05:42.046440 (+ 2422us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.046454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:42.046543 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:42.047288 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:42.047294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:42.047321 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:05:42.047672 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.047679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.059153 (+ 11474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.059180 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:42.060063 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.060073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.060171 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.060189 (+ 18us) client_negotiation.cc:769] Sending connection context
0202 15:05:42.060231 (+ 42us) client_negotiation.cc:240] Negotiation successful
0202 15:05:42.060296 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"thread_start_us":80,"threads_started":1}
I20260202 15:05:42.060312 14790 tablet_service.cc:1505] Processing CreateTablet for tablet 51050225a0b44a0bacd41ac8e70f02cd (DEFAULT_TABLE table=test-table [id=ddccb485a59d400ea13055eb76f0124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:05:42.061165 14790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51050225a0b44a0bacd41ac8e70f02cd. 1 dirs total, 0 dirs full, 0 dirs failed
Feb 02 15:05:42 dist-test-slave-6zzp krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:05:42.061880 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.044689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.2.164.66:41945)
0202 15:05:42.046795 (+ 2106us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.046802 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.046826 (+ 24us) server_negotiation.cc:408] Connection header received
0202 15:05:42.046909 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.046914 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.047100 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.057083 (+ 9983us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:05:42.057248 (+ 165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.058937 (+ 1689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.060792 (+ 1855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.061480 (+ 688us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.061591 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.061660 (+ 69us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.061749 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":114,"threads_started":1}
I20260202 15:05:42.064545 14928 tablet_service.cc:1505] Processing CreateTablet for tablet 51050225a0b44a0bacd41ac8e70f02cd (DEFAULT_TABLE table=test-table [id=ddccb485a59d400ea13055eb76f0124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260202 15:05:42.065105 15149 tablet_bootstrap.cc:492] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Bootstrap starting.
I20260202 15:05:42.065497 14928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51050225a0b44a0bacd41ac8e70f02cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:05:42.067463 15149 tablet_bootstrap.cc:654] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Neither blocks nor log segments found. Creating new log.
I20260202 15:05:42.068825 15149 log.cc:826] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Log is configured to *not* fsync() on all Append() calls
I20260202 15:05:42.070812 15149 tablet_bootstrap.cc:492] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: No bootstrap required, opened a new log
I20260202 15:05:42.071003 15149 ts_tablet_manager.cc:1403] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:05:42.073760 15151 tablet_bootstrap.cc:492] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: Bootstrap starting.
I20260202 15:05:42.074914 15149 raft_consensus.cc:359] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.075258 15149 raft_consensus.cc:385] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:05:42.075354 15149 raft_consensus.cc:740] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc8e4c7526347b6b6a669484d72c987, State: Initialized, Role: FOLLOWER
I20260202 15:05:42.075994 15149 consensus_queue.cc:260] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.076130 15151 tablet_bootstrap.cc:654] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: Neither blocks nor log segments found. Creating new log.
I20260202 15:05:42.077085 15149 ts_tablet_manager.cc:1434] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:05:42.077178 15151 log.cc:826] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: Log is configured to *not* fsync() on all Append() calls
I20260202 15:05:42.077503 15130 heartbeater.cc:499] Master 127.2.164.126:46089 was elected leader, sending a full tablet report...
I20260202 15:05:42.079128 15152 tablet_bootstrap.cc:492] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: Bootstrap starting.
I20260202 15:05:42.079947 15151 tablet_bootstrap.cc:492] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: No bootstrap required, opened a new log
I20260202 15:05:42.080148 15151 ts_tablet_manager.cc:1403] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260202 15:05:42.082471 15152 tablet_bootstrap.cc:654] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: Neither blocks nor log segments found. Creating new log.
I20260202 15:05:42.083760 15152 log.cc:826] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: Log is configured to *not* fsync() on all Append() calls
I20260202 15:05:42.084259 15151 raft_consensus.cc:359] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.084555 15151 raft_consensus.cc:385] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:05:42.084657 15151 raft_consensus.cc:740] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e16181a211a4461905e6d8a841afd94, State: Initialized, Role: FOLLOWER
I20260202 15:05:42.085424 15151 consensus_queue.cc:260] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.086148 14856 heartbeater.cc:499] Master 127.2.164.126:46089 was elected leader, sending a full tablet report...
I20260202 15:05:42.086361 15151 ts_tablet_manager.cc:1434] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260202 15:05:42.090310 15152 tablet_bootstrap.cc:492] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: No bootstrap required, opened a new log
I20260202 15:05:42.090598 15152 ts_tablet_manager.cc:1403] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260202 15:05:42.094067 15152 raft_consensus.cc:359] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.094406 15152 raft_consensus.cc:385] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:05:42.094508 15152 raft_consensus.cc:740] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4d60df67834be1a9d7a53387d93537, State: Initialized, Role: FOLLOWER
I20260202 15:05:42.095074 15152 consensus_queue.cc:260] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.096172 14994 heartbeater.cc:499] Master 127.2.164.126:46089 was elected leader, sending a full tablet report...
I20260202 15:05:42.096187 15152 ts_tablet_manager.cc:1434] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260202 15:05:42.141067 14995 tablet.cc:2378] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:05:42.208318 15153 raft_consensus.cc:493] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260202 15:05:42.208570 15153 raft_consensus.cc:515] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.209975 15153 leader_election.cc:290] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a4d60df67834be1a9d7a53387d93537 (127.2.164.66:41945), 3e16181a211a4461905e6d8a841afd94 (127.2.164.65:35983)
I20260202 15:05:42.214089 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.210369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41945 (local address 127.2.164.67:46593)
0202 15:05:42.210688 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.210704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:42.210808 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:42.211336 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:42.211340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:42.211358 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:05:42.211635 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.211643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.212697 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.212701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:42.213653 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.213665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.213806 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.213831 (+ 25us) client_negotiation.cc:769] Sending connection context
0202 15:05:42.213890 (+ 59us) client_negotiation.cc:240] Negotiation successful
0202 15:05:42.213956 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260202 15:05:42.215061 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.210708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:46593 (local address 127.2.164.66:41945)
0202 15:05:42.210836 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.210844 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.210878 (+ 34us) server_negotiation.cc:408] Connection header received
0202 15:05:42.210936 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.210939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.210998 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.211070 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:05:42.211780 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.212569 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.214287 (+ 1718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.214794 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.214834 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.214886 (+ 52us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.214945 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260202 15:05:42.216089 14940 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51050225a0b44a0bacd41ac8e70f02cd" candidate_uuid: "8cc8e4c7526347b6b6a669484d72c987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a4d60df67834be1a9d7a53387d93537" is_pre_election: true
I20260202 15:05:42.216387 14940 raft_consensus.cc:2468] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc8e4c7526347b6b6a669484d72c987 in term 0.
I20260202 15:05:42.216787 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.211241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:33805 (local address 127.2.164.65:35983)
0202 15:05:42.211608 (+ 367us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.211612 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.212897 (+ 1285us) server_negotiation.cc:408] Connection header received
0202 15:05:42.213111 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.213115 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.213170 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.213220 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0202 15:05:42.213852 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.214625 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.215851 (+ 1226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.216346 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.216394 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.216600 (+ 206us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.216671 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260202 15:05:42.217060 15017 leader_election.cc:304] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a4d60df67834be1a9d7a53387d93537, 8cc8e4c7526347b6b6a669484d72c987; no voters:
I20260202 15:05:42.217180 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.211101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35983 (local address 127.2.164.67:33805)
0202 15:05:42.212818 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.212833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0202 15:05:42.212960 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:05:42.213352 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:05:42.213355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:05:42.213373 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0202 15:05:42.213702 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.213711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.214769 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.214773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:05:42.215706 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:05:42.215716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.216454 (+ 738us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.216470 (+ 16us) client_negotiation.cc:769] Sending connection context
0202 15:05:42.216909 (+ 439us) client_negotiation.cc:240] Negotiation successful
0202 15:05:42.216972 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1640,"thread_start_us":103,"threads_started":1}
I20260202 15:05:42.217398 15153 raft_consensus.cc:2804] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260202 15:05:42.217468 15153 raft_consensus.cc:493] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260202 15:05:42.217514 15153 raft_consensus.cc:3060] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:05:42.218750 15153 raft_consensus.cc:515] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.219142 15153 leader_election.cc:290] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [CANDIDATE]: Term 1 election: Requested vote from peers 7a4d60df67834be1a9d7a53387d93537 (127.2.164.66:41945), 3e16181a211a4461905e6d8a841afd94 (127.2.164.65:35983)
I20260202 15:05:42.219722 14795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51050225a0b44a0bacd41ac8e70f02cd" candidate_uuid: "8cc8e4c7526347b6b6a669484d72c987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e16181a211a4461905e6d8a841afd94" is_pre_election: true
I20260202 15:05:42.219964 14795 raft_consensus.cc:2468] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc8e4c7526347b6b6a669484d72c987 in term 0.
I20260202 15:05:42.220326 14808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51050225a0b44a0bacd41ac8e70f02cd" candidate_uuid: "8cc8e4c7526347b6b6a669484d72c987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e16181a211a4461905e6d8a841afd94"
I20260202 15:05:42.220430 14808 raft_consensus.cc:3060] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:05:42.220700 14940 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51050225a0b44a0bacd41ac8e70f02cd" candidate_uuid: "8cc8e4c7526347b6b6a669484d72c987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a4d60df67834be1a9d7a53387d93537"
I20260202 15:05:42.220813 14940 raft_consensus.cc:3060] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:05:42.221696 14808 raft_consensus.cc:2468] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc8e4c7526347b6b6a669484d72c987 in term 1.
I20260202 15:05:42.222034 14940 raft_consensus.cc:2468] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc8e4c7526347b6b6a669484d72c987 in term 1.
I20260202 15:05:42.222437 15020 leader_election.cc:304] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e16181a211a4461905e6d8a841afd94, 8cc8e4c7526347b6b6a669484d72c987; no voters:
I20260202 15:05:42.222894 15153 raft_consensus.cc:2804] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:05:42.223089 15153 raft_consensus.cc:697] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc8e4c7526347b6b6a669484d72c987, State: Running, Role: LEADER
I20260202 15:05:42.223402 15153 consensus_queue.cc:237] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } }
I20260202 15:05:42.227061 14641 catalog_manager.cc:5654] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cc8e4c7526347b6b6a669484d72c987 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "8cc8e4c7526347b6b6a669484d72c987" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc8e4c7526347b6b6a669484d72c987" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 } health_report { overall_health: UNKNOWN } } }
W20260202 15:05:42.237185 15131 tablet.cc:2378] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:05:42.258929 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.254514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.2.164.67:43181)
0202 15:05:42.254642 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0202 15:05:42.254646 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0202 15:05:42.254661 (+ 15us) server_negotiation.cc:408] Connection header received
0202 15:05:42.254756 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:05:42.254759 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:05:42.254836 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:05:42.254939 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
0202 15:05:42.255418 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.256172 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:05:42.257256 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:05:42.257473 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:05:42.257592 (+ 119us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0202 15:05:42.258190 (+ 598us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0202 15:05:42.258311 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0202 15:05:42.258727 (+ 416us) server_negotiation.cc:300] Negotiation successful
0202 15:05:42.258785 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260202 15:05:42.265975 14808 raft_consensus.cc:1275] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94 [term 1 FOLLOWER]: Refusing update from remote peer 8cc8e4c7526347b6b6a669484d72c987: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:05:42.266047 14940 raft_consensus.cc:1275] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 1 FOLLOWER]: Refusing update from remote peer 8cc8e4c7526347b6b6a669484d72c987: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260202 15:05:42.266744 15160 consensus_queue.cc:1048] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a4d60df67834be1a9d7a53387d93537" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260202 15:05:42.266999 15153 consensus_queue.cc:1048] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e16181a211a4461905e6d8a841afd94" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260202 15:05:42.270665 14857 tablet.cc:2378] T 51050225a0b44a0bacd41ac8e70f02cd P 3e16181a211a4461905e6d8a841afd94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260202 15:05:42.283732 15163 mvcc.cc:204] Tried to move back new op lower bound from 7250103264312893440 to 7250103264150908928. Current Snapshot: MvccSnapshot[applied={T|T < 7250103264312893440}]
I20260202 15:05:42.287506 15167 mvcc.cc:204] Tried to move back new op lower bound from 7250103264312893440 to 7250103264150908928. Current Snapshot: MvccSnapshot[applied={T|T < 7250103264312893440}]
I20260202 15:05:42.287869 15165 mvcc.cc:204] Tried to move back new op lower bound from 7250103264312893440 to 7250103264150908928. Current Snapshot: MvccSnapshot[applied={T|T < 7250103264312893440}]
I20260202 15:05:42.299088 14641 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:05:42.299327 14641 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260202 15:05:42.302150 14641 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 66b44d1bb77342579b44a9b25a537a88: Sending DeleteTablet for 3 replicas of tablet 51050225a0b44a0bacd41ac8e70f02cd
I20260202 15:05:42.303471 14928 tablet_service.cc:1552] Processing DeleteTablet for tablet 51050225a0b44a0bacd41ac8e70f02cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:05:42 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:52732
I20260202 15:05:42.304041 14790 tablet_service.cc:1552] Processing DeleteTablet for tablet 51050225a0b44a0bacd41ac8e70f02cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:05:42 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:46914
I20260202 15:05:42.305027 15064 tablet_service.cc:1552] Processing DeleteTablet for tablet 51050225a0b44a0bacd41ac8e70f02cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-02 15:05:42 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:44860
I20260202 15:05:42.306205 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 14723
I20260202 15:05:42.310967 15176 tablet_replica.cc:333] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: stopping tablet replica
I20260202 15:05:42.311306 15176 raft_consensus.cc:2243] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 1 LEADER]: Raft consensus shutting down.
I20260202 15:05:42.311853 15176 raft_consensus.cc:2272] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:05:42.312145 15174 tablet_replica.cc:333] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537: stopping tablet replica
I20260202 15:05:42.313268 15174 raft_consensus.cc:2243] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260202 15:05:42.313935 15174 raft_consensus.cc:2272] T 51050225a0b44a0bacd41ac8e70f02cd P 7a4d60df67834be1a9d7a53387d93537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260202 15:05:42.315424 15176 ts_tablet_manager.cc:1916] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260202 15:05:42.320755 14628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:35983 (error 108)
I20260202 15:05:42.321692 15176 ts_tablet_manager.cc:1929] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260202 15:05:42.321799 15176 log.cc:1199] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Deleting WAL directory at /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770044596670313-2705-0/minicluster-data/ts-2/wal/wals/51050225a0b44a0bacd41ac8e70f02cd
I20260202 15:05:42.322052 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 14864
I20260202 15:05:42.325348 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.324956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35983 (local address 127.0.0.1:46918)
0202 15:05:42.325182 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.325251 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260202 15:05:42.325640 14628 catalog_manager.cc:4712] TS 3e16181a211a4461905e6d8a841afd94 (127.2.164.65:35983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51050225a0b44a0bacd41ac8e70f02cd: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35983: connect: Connection refused (error 111)
I20260202 15:05:42.329221 15176 ts_tablet_manager.cc:1950] T 51050225a0b44a0bacd41ac8e70f02cd P 8cc8e4c7526347b6b6a669484d72c987: Deleting consensus metadata
I20260202 15:05:42.330727 14626 catalog_manager.cc:4985] TS 8cc8e4c7526347b6b6a669484d72c987 (127.2.164.67:43181): tablet 51050225a0b44a0bacd41ac8e70f02cd (table test-table [id=ddccb485a59d400ea13055eb76f0124c]) successfully deleted
I20260202 15:05:42.331825 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 15001
I20260202 15:05:42.336717 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:05:42.336331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41945 (local address 127.0.0.1:52748)
0202 15:05:42.336560 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0202 15:05:42.336613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.66:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260202 15:05:42.339828 2705 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu with pid 14609
I20260202 15:05:42.348649 2705 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260202 15:05:42.694381 2705 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260202 15:05:42.694573 2705 mini_ranger.cc:66] Stopping Ranger...
I20260202 15:05:43.040477 2705 mini_ranger.cc:68] Stopped Ranger
2026-02-02 15:05:43.040 UTC [14169] LOG: received smart shutdown request
2026-02-02 15:05:43.044 UTC [14169] LOG: background worker "logical replication launcher" (PID 14177) exited with exit code 1
2026-02-02 15:05:43.045 UTC [14172] LOG: shutting down
2026-02-02 15:05:43.047 UTC [14172] LOG: checkpoint starting: shutdown immediate
2026-02-02 15:05:44.062 UTC [14172] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.036 s, sync=0.973 s, total=1.018 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/20450F0, redo lsn=0/20450F0
2026-02-02 15:05:44.071 UTC [14169] LOG: database system is shut down
2026-02-02T15:05:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49342 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 02 15:05:44 dist-test-slave-6zzp krb5kdc[15187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 02 15:05:44 dist-test-slave-6zzp krb5kdc[15187](info): set up 2 sockets
Feb 02 15:05:44 dist-test-slave-6zzp krb5kdc[15187](info): commencing operation
krb5kdc: starting...
W20260202 15:05:46.217386 2705 mini_kdc.cc:121] Time spent starting KDC: real 2.001s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/test-user.keytab.
Feb 02 15:05:46 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-02T15:05:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-02T15:05:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
I20260202 15:05:46.383200 2705 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRqWm_m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/postgres -l logfile start
2026-02-02 15:05:49.145 UTC [15218] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-02 15:05:49.145 UTC [15218] LOG: listening on IPv4 address "127.2.164.84", port 38795
2026-02-02 15:05:49.149 UTC [15218] LOG: listening on Unix socket "/tmp/.s.PGSQL.38795"
2026-02-02 15:05:49.155 UTC [15223] LOG: database system was shut down at 2026-02-02 15:05:47 UTC
2026-02-02 15:05:49.159 UTC [15218] LOG: database system is ready to accept connections
I20260202 15:05:51.069172 2705 mini_postgres.cc:96] Postgres bound to 38795
2026-02-02 15:05:51.075 UTC [15229] FATAL: database "slave" does not exist
127.2.164.84:38795 - accepting connections
I20260202 15:05:51.076256 2705 mini_ranger.cc:161] Starting Ranger...
I20260202 15:05:51.094763 2705 mini_ranger.cc:84] Created miniranger Postgres user
I20260202 15:05:51.165143 2705 mini_ranger.cc:87] Created ranger Postgres database
I20260202 15:05:51.165242 2705 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-admin
2026-02-02 15:05:51,434 [I] DB FLAVOR :POSTGRES
2026-02-02 15:05:51,435 [I] --------- Verifying Ranger DB connection ---------
2026-02-02 15:05:51,435 [I] Checking connection..
2026-02-02 15:05:51,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-02 15:05:51,752 [I] Checking connection passed.
2026-02-02 15:05:51,753 [I] --------- Verifying version history table ---------
2026-02-02 15:05:51,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-02 15:05:52,059 [I] Table x_db_version_h does not exist in database ranger
2026-02-02 15:05:52,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-02 15:05:52,393 [I] Table x_db_version_h does not exist in database ranger
2026-02-02 15:05:52,394 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-02 15:05:52,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-02 15:05:52.702 UTC [15296] WARNING: there is no transaction in progress
2026-02-02 15:05:52,710 [I] create_dbversion_catalog.sql file imported successfully
2026-02-02 15:05:52,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-02 15:05:53,019 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-02 15:05:53,019 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-02 15:05:53,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-02 15:05:53,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-02 15:05:53,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6zzp.c.gcp-upstream.internal','N') ;"
2026-02-02 15:05:53,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-02 15:05:54,238 [I] Table x_portal_user does not exist in database ranger
2026-02-02 15:05:54,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-02 15:05:54,554 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-02 15:05:54,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-02 15:05:54,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-02 15:05:55,173 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-02 15:05:55,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-02 15:05:56.095 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.107 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.121 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.132 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.143 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.154 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.202 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.208 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.219 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.228 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.239 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.251 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.260 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.265 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.268 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56.272 UTC [15413] WARNING: there is no transaction in progress
2026-02-02 15:05:56,704 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-02 15:05:56,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-02 15:05:57,003 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-02 15:05:57,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-02 15:05:57,343 [I] Table x_portal_user already exists in database 'ranger'
2026-02-02 15:05:57,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-02 15:05:57,645 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-02 15:05:57,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-02 15:05:57,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-02 15:05:58,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6zzp.c.gcp-upstream.internal';"
2026-02-02 15:05:58,564 [I] CORE_DB_SCHEMA import status has been updated
2026-02-02 15:05:58,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-02 15:05:58,876 [I] DB_PATCHES have already been applied
I20260202 15:05:58.881788 2705 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRqWm_m/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260202 15:05:58.881865 2705 mini_ranger.cc:193] Using host: 127.2.164.84
I20260202 15:05:58.884843 2705 mini_ranger.cc:227] Ranger admin URL: http://127.2.164.84:53023
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-admin/ews/webapp].
Feb 02, 2026 3:05:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260202 15:06:01.152115 2705 mini_ranger.cc:160] Time spent starting Ranger: real 10.076s user 0.000s sys 0.011s
Feb 02, 2026 3:06:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 02, 2026 3:06:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Feb 02, 2026 3:06:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 02, 2026 3:06:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 02, 2026 3:06:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260202 15:06:23.034355 2705 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.164.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 02 15:06:23 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044783, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260202 15:06:23.133850 2705 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260202 15:06:23.144608 2705 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260202 15:06:23.252844 2705 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260202 15:06:23.252943 2705 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms
I20260202 15:06:23.252960 2705 mini_ranger_kms.cc:227] Using postgres at 127.2.164.84:38795
2026-02-02 15:06:23,456 [I] DB FLAVOR :POSTGRES
2026-02-02 15:06:23,457 [I] --------- Verifying Ranger DB connection ---------
2026-02-02 15:06:23,457 [I] Checking connection
2026-02-02 15:06:23,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-02 15:06:23,783 [I] connection success
2026-02-02 15:06:23,783 [I] --------- Verifying Ranger DB tables ---------
2026-02-02 15:06:23,783 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-02 15:06:23,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-02 15:06:24,108 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-02 15:06:24,108 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-02 15:06:24,108 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-02 15:06:24,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:38795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-02 15:06:24,413 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260202 15:06:24.895599 2705 mini_ranger_kms.cc:310] Created kms service
I20260202 15:06:25.104029 2705 mini_ranger_kms.cc:326] Created kudu user
I20260202 15:06:25.142849 2705 mini_ranger_kms.cc:343] Created rangerkms user
Feb 02 15:06:25 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044783, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:06:25.513689 2705 mini_ranger_kms.cc:384] Added ranger policy
I20260202 15:06:25.513921 2705 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRqWm_m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRqWm_m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260202 15:06:25.513948 2705 mini_ranger_kms.cc:240] Using host: 127.2.164.84
I20260202 15:06:25.517046 2705 mini_ranger_kms.cc:276] Ranger KMS PID: 15605
I20260202 15:06:25.517135 2705 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.164.84:48233
Feb 02, 2026 3:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 02, 2026 3:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 02, 2026 3:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 02, 2026 3:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 02, 2026 3:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-kms/ews/webapp].
Feb 02, 2026 3:06:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260202 15:06:27.721791 2705 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.588s user 0.007s sys 0.180s
I20260202 15:06:27.721923 2705 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260202 15:06:27.721999 2705 mini_ranger_kms.cc:401] 127.2.164.84:48233/kms/v1/keys
Feb 02 15:06:27 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1770044787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 02 15:06:28 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Feb 02 15:06:28 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Feb 02 15:06:30 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:06:30.750133 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40587
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40587
--ranger_config_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:48233/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:06:30.859576 15658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:06:30.859794 15658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:06:30.859867 15658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:06:30.863116 15658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260202 15:06:30.863171 15658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:06:30.863194 15658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:06:30.863212 15658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260202 15:06:30.863229 15658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260202 15:06:30.867576 15658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.164.84:48233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40587
--ranger_config_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:06:30.868564 15658 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:06:30.869441 15658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:30.874825 15663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:30.874848 15666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:30.874948 15658 server_base.cc:1060] running on GCE node
W20260202 15:06:30.875154 15664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:30.875669 15658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:06:30.876559 15658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:06:30.877739 15658 hybrid_clock.cc:648] HybridClock initialized: now 1770044790877720 us; error 39 us; skew 500 ppm
Feb 02 15:06:30 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044790, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:06:30.880484 15658 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260202 15:06:30.881701 15658 webserver.cc:492] Webserver started at http://127.2.164.126:40943/ using document root <none> and password file <none>
I20260202 15:06:30.882297 15658 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:30.882352 15658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:30.882560 15658 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:06:30 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044790, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:06:30.929955 15658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data/instance:
uuid: "bd86909b2e364d60a4890faaecd967be"
format_stamp: "Formatted at 2026-02-02 15:06:30 on dist-test-slave-6zzp"
server_key: "e7c66321628d4a62c7d5397deafbb828"
server_key_iv: "11e6341033bb4325e8c0f3989f770668"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:30.930516 15658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal/instance:
uuid: "bd86909b2e364d60a4890faaecd967be"
format_stamp: "Formatted at 2026-02-02 15:06:30 on dist-test-slave-6zzp"
server_key: "e7c66321628d4a62c7d5397deafbb828"
server_key_iv: "11e6341033bb4325e8c0f3989f770668"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:30.934149 15658 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260202 15:06:30.948552 15673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:30.949793 15658 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260202 15:06:30.949937 15658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal
uuid: "bd86909b2e364d60a4890faaecd967be"
format_stamp: "Formatted at 2026-02-02 15:06:30 on dist-test-slave-6zzp"
server_key: "e7c66321628d4a62c7d5397deafbb828"
server_key_iv: "11e6341033bb4325e8c0f3989f770668"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:30.950079 15658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:30.971314 15658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:30.978881 15658 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:06:30.979128 15658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:30.988297 15658 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40587
I20260202 15:06:30.988314 15725 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.126:40587 every 8 connection(s)
I20260202 15:06:30.989472 15658 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/data/info.pb
I20260202 15:06:30.992357 15726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260202 15:06:30.997231 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 15658
I20260202 15:06:30.997385 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/wal/instance
I20260202 15:06:30.997618 2705 external_mini_cluster.cc:1468] Setting key cdec490b48a76048edff1357c0d19202
I20260202 15:06:30.998188 15726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be: Bootstrap starting.
I20260202 15:06:31.001144 15726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be: Neither blocks nor log segments found. Creating new log.
I20260202 15:06:31.002009 15726 log.cc:826] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be: Log is configured to *not* fsync() on all Append() calls
I20260202 15:06:31.004340 15726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be: No bootstrap required, opened a new log
Feb 02 15:06:31 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:06:31.007925 15726 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd86909b2e364d60a4890faaecd967be" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40587 } }
I20260202 15:06:31.008190 15726 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260202 15:06:31.008322 15726 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd86909b2e364d60a4890faaecd967be, State: Initialized, Role: FOLLOWER
I20260202 15:06:31.008885 15726 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd86909b2e364d60a4890faaecd967be" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40587 } }
I20260202 15:06:31.009112 15726 raft_consensus.cc:399] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260202 15:06:31.009217 15726 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260202 15:06:31.009347 15726 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 0 FOLLOWER]: Advancing to term 1
I20260202 15:06:31.010377 15726 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd86909b2e364d60a4890faaecd967be" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40587 } }
I20260202 15:06:31.010771 15726 leader_election.cc:304] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd86909b2e364d60a4890faaecd967be; no voters:
I20260202 15:06:31.011176 15726 leader_election.cc:290] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260202 15:06:31.011270 15731 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 1 FOLLOWER]: Leader election won for term 1
I20260202 15:06:31.011511 15731 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [term 1 LEADER]: Becoming Leader. State: Replica: bd86909b2e364d60a4890faaecd967be, State: Running, Role: LEADER
I20260202 15:06:31.012059 15731 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd86909b2e364d60a4890faaecd967be" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40587 } }
I20260202 15:06:31.012533 15726 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [sys.catalog]: configured and running, proceeding with master startup.
I20260202 15:06:31.013173 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:30.998991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.2.164.126:40587)
0202 15:06:30.999480 (+ 489us) server_negotiation.cc:207] Beginning negotiation
0202 15:06:30.999495 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0202 15:06:30.999542 (+ 47us) server_negotiation.cc:408] Connection header received
0202 15:06:31.000307 (+ 765us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:06:31.000332 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:06:31.000696 (+ 364us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:06:31.001104 (+ 408us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:06:31.002191 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:06:31.003157 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:06:31.004475 (+ 1318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:06:31.006078 (+ 1603us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:06:31.007696 (+ 1618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:06:31.007719 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:06:31.007734 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:06:31.007774 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:06:31.009835 (+ 2061us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:06:31.010362 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:06:31.010368 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:06:31.010373 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:06:31.010449 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:06:31.010756 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:06:31.010759 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:06:31.010761 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:06:31.011166 (+ 405us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:06:31.011943 (+ 777us) server_negotiation.cc:1036] Waiting for connection context
0202 15:06:31.012104 (+ 161us) server_negotiation.cc:300] Negotiation successful
0202 15:06:31.012334 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":152,"threads_started":1}
I20260202 15:06:31.013543 15731 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd86909b2e364d60a4890faaecd967be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd86909b2e364d60a4890faaecd967be" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40587 } } }
I20260202 15:06:31.013756 15731 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [sys.catalog]: This master's current role is: LEADER
I20260202 15:06:31.014019 15731 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd86909b2e364d60a4890faaecd967be. Latest consensus state: current_term: 1 leader_uuid: "bd86909b2e364d60a4890faaecd967be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd86909b2e364d60a4890faaecd967be" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40587 } } }
I20260202 15:06:31.014166 15731 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be [sys.catalog]: This master's current role is: LEADER
I20260202 15:06:31.014516 15740 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260202 15:06:31.019290 15740 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260202 15:06:31.019757 15726 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260202 15:06:31.034030 15740 catalog_manager.cc:1357] Generated new cluster ID: 60feb67ac4454127a154533fb6f5f99b
I20260202 15:06:31.034137 15740 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260202 15:06:31.042805 15740 catalog_manager.cc:1380] Generated new certificate authority record
I20260202 15:06:31.044196 15740 catalog_manager.cc:1514] Loading token signing keys...
I20260202 15:06:31.055931 15740 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bd86909b2e364d60a4890faaecd967be: Generated new TSK 0
I20260202 15:06:31.056973 15740 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:06:31.895898 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:48233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40587
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260202 15:06:32.024181 15772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:06:32.024472 15772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:06:32.024613 15772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:06:32.028268 15772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:06:32.028343 15772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:06:32.028463 15772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260202 15:06:32.033308 15772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.164.84:48233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:06:32.034423 15772 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:06:32.035298 15772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:32.041857 15779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:32.042001 15782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:32.041857 15780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:32.042615 15772 server_base.cc:1060] running on GCE node
I20260202 15:06:32.043038 15772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:06:32.043732 15772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:06:32.044930 15772 hybrid_clock.cc:648] HybridClock initialized: now 1770044792044897 us; error 42 us; skew 500 ppm
Feb 02 15:06:32 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:06:32.048326 15772 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260202 15:06:32.049638 15772 webserver.cc:492] Webserver started at http://127.2.164.65:46697/ using document root <none> and password file <none>
I20260202 15:06:32.050313 15772 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:32.050377 15772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:32.050575 15772 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:06:32 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:06:32.067178 15772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data/instance:
uuid: "cfc6fe6f75594ab58456965a01e28213"
format_stamp: "Formatted at 2026-02-02 15:06:32 on dist-test-slave-6zzp"
server_key: "eb05161e3412a0b04fffacc6a2279ff8"
server_key_iv: "e5ba0aa6f940779a2cda718f77629710"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:32.068101 15772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "cfc6fe6f75594ab58456965a01e28213"
format_stamp: "Formatted at 2026-02-02 15:06:32 on dist-test-slave-6zzp"
server_key: "eb05161e3412a0b04fffacc6a2279ff8"
server_key_iv: "e5ba0aa6f940779a2cda718f77629710"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:32.073164 15772 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260202 15:06:32.093411 15789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:32.094846 15772 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260202 15:06:32.095011 15772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal
uuid: "cfc6fe6f75594ab58456965a01e28213"
format_stamp: "Formatted at 2026-02-02 15:06:32 on dist-test-slave-6zzp"
server_key: "eb05161e3412a0b04fffacc6a2279ff8"
server_key_iv: "e5ba0aa6f940779a2cda718f77629710"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:32.095126 15772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:32.115059 15772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:32.118741 15772 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:06:32.119011 15772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:32.119760 15772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:06:32.120950 15772 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:06:32.121089 15772 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:32.121233 15772 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:06:32.121326 15772 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:32.132752 15772 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:41817
I20260202 15:06:32.133056 15904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.65:41817 every 8 connection(s)
I20260202 15:06:32.133881 15772 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/data/info.pb
I20260202 15:06:32.136189 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 15772
I20260202 15:06:32.136324 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-0/wal/instance
I20260202 15:06:32.136584 2705 external_mini_cluster.cc:1468] Setting key c12f3c341e388a9a65d586ec880db5d2
Feb 02 15:06:32 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260202 15:06:32.147949 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:32.135806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34645 (local address 127.2.164.126:40587)
0202 15:06:32.136064 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0202 15:06:32.136069 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0202 15:06:32.136796 (+ 727us) server_negotiation.cc:408] Connection header received
0202 15:06:32.137940 (+ 1144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0202 15:06:32.137944 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0202 15:06:32.138002 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0202 15:06:32.138062 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0202 15:06:32.139775 (+ 1713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:06:32.140648 (+ 873us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0202 15:06:32.141536 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0202 15:06:32.141724 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:06:32.144393 (+ 2669us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0202 15:06:32.144414 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0202 15:06:32.144417 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0202 15:06:32.144451 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0202 15:06:32.145984 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:06:32.146495 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:06:32.146500 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:06:32.146502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:06:32.146559 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0202 15:06:32.146871 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0202 15:06:32.146875 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0202 15:06:32.146877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0202 15:06:32.147027 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0202 15:06:32.147134 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0202 15:06:32.147640 (+ 506us) server_negotiation.cc:300] Negotiation successful
0202 15:06:32.147765 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260202 15:06:32.148712 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:32.136129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.65:34645)
0202 15:06:32.136608 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:32.136707 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0202 15:06:32.137708 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:06:32.138475 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0202 15:06:32.138485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0202 15:06:32.138834 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0202 15:06:32.139605 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:06:32.139622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:06:32.140806 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0202 15:06:32.140810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0202 15:06:32.141400 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0202 15:06:32.141410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0202 15:06:32.141970 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0202 15:06:32.142628 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0202 15:06:32.142664 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0202 15:06:32.144237 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0202 15:06:32.146126 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:06:32.146132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:06:32.146143 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:06:32.146393 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:06:32.146671 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0202 15:06:32.146674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0202 15:06:32.146675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0202 15:06:32.146776 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0202 15:06:32.147134 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0202 15:06:32.147142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0202 15:06:32.147387 (+ 245us) client_negotiation.cc:769] Sending connection context
0202 15:06:32.147595 (+ 208us) client_negotiation.cc:240] Negotiation successful
0202 15:06:32.147842 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260202 15:06:32.153330 15905 heartbeater.cc:344] Connected to a master server at 127.2.164.126:40587
I20260202 15:06:32.153630 15905 heartbeater.cc:461] Registering TS with master...
I20260202 15:06:32.154306 15905 heartbeater.cc:507] Master 127.2.164.126:40587 requested a full tablet report, sending...
I20260202 15:06:32.156194 15690 ts_manager.cc:194] Registered new tserver with Master: cfc6fe6f75594ab58456965a01e28213 (127.2.164.65:41817)
I20260202 15:06:32.157868 15690 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:34645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab.
I20260202 15:06:32.201172 2705 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
/tmp/dist-test-taskRqWm_m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:48233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40587
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.2.164.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.2.164.126 from keytab /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20260202 15:06:32.223825 15764 server.cc:273] Received an EOF from the subprocess
F20260202 15:06:32.224858 15751 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fa8f2441dcd google::LogMessage::Fail() at ??:0
@ 0x7fa8f2445b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa8f24417cc google::LogMessage::Flush() at ??:0
@ 0x7fa8f2442f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa8f40f92d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fa8f40f60d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fa8f40fa005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55dd1a249a0a std::function<>::operator()() at ??:0
@ 0x7fa8f389bb04 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa8f41356db start_thread at ??:0
@ 0x7fa8f15f871f clone at ??:0
W20260202 15:06:32.314971 15913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260202 15:06:32.315234 15913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260202 15:06:32.315325 15913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260202 15:06:32.318748 15913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260202 15:06:32.318835 15913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260202 15:06:32.318941 15913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260202 15:06:32.323098 15913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.164.84:48233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/krb5kdc/kudu.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.2.164.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 02 Feb 2026 14:43:13 UTC on 24a791456cd2
build id 10310
I20260202 15:06:32.324142 15913 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260202 15:06:32.324991 15913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260202 15:06:32.331635 15930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260202 15:06:32.331600 15933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:32.331911 15913 server_base.cc:1060] running on GCE node
W20260202 15:06:32.331928 15931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260202 15:06:32.332304 15913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260202 15:06:32.332969 15913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260202 15:06:32.334192 15913 hybrid_clock.cc:648] HybridClock initialized: now 1770044792334166 us; error 37 us; skew 500 ppm
Feb 02 15:06:32 dist-test-slave-6zzp krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260202 15:06:32.337467 15913 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260202 15:06:32.338729 15913 webserver.cc:492] Webserver started at http://127.2.164.66:36321/ using document root <none> and password file <none>
I20260202 15:06:32.339416 15913 fs_manager.cc:362] Metadata directory not provided
I20260202 15:06:32.339473 15913 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260202 15:06:32.339632 15913 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 02 15:06:32 dist-test-slave-6zzp krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770044792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260202 15:06:32.353730 15913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data/instance:
uuid: "b5520805befb45038c269ef1e71fc0b7"
format_stamp: "Formatted at 2026-02-02 15:06:32 on dist-test-slave-6zzp"
server_key: "f554399f60c54ca6e137bb92cb6f41fc"
server_key_iv: "bbafb0242b3f03a1d474a5cbe3f91001"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:32.354280 15913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "b5520805befb45038c269ef1e71fc0b7"
format_stamp: "Formatted at 2026-02-02 15:06:32 on dist-test-slave-6zzp"
server_key: "f554399f60c54ca6e137bb92cb6f41fc"
server_key_iv: "bbafb0242b3f03a1d474a5cbe3f91001"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:32.358034 15913 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260202 15:06:32.373348 15940 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:32.374519 15913 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260202 15:06:32.374650 15913 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal
uuid: "b5520805befb45038c269ef1e71fc0b7"
format_stamp: "Formatted at 2026-02-02 15:06:32 on dist-test-slave-6zzp"
server_key: "f554399f60c54ca6e137bb92cb6f41fc"
server_key_iv: "bbafb0242b3f03a1d474a5cbe3f91001"
server_key_version: "kuduclusterkey@0"
I20260202 15:06:32.374781 15913 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260202 15:06:32.404150 15913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260202 15:06:32.407675 15913 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260202 15:06:32.407927 15913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260202 15:06:32.408653 15913 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260202 15:06:32.409816 15913 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260202 15:06:32.409920 15913 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:32.410015 15913 ts_tablet_manager.cc:616] Registered 0 tablets
I20260202 15:06:32.410076 15913 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260202 15:06:32.421665 15913 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:46127
I20260202 15:06:32.421938 16053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.164.66:46127 every 8 connection(s)
I20260202 15:06:32.422927 15913 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/data/info.pb
I20260202 15:06:32.428053 2705 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRqWm_m/build/debug/bin/kudu as pid 15913
I20260202 15:06:32.428177 2705 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRqWm_m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770044596670313-2705-0/minicluster-data/ts-1/wal/instance
I20260202 15:06:32.428439 2705 external_mini_cluster.cc:1468] Setting key df7e13b54aef668ccb1d91b8e1456bd6
W20260202 15:06:32.467085 15185 connection.cc:537] client connection to 127.2.164.126:40587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260202 15:06:32.468194 16056 negotiation.cc:337] Failed RPC negotiation. Trace:
0202 15:06:32.425327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:46783)
0202 15:06:32.425883 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:32.425928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0202 15:06:32.426892 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0202 15:06:32.467505 (+ 40613us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":133,"threads_started":1}
W20260202 15:06:32.469054 16054 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:40587 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:40587: Client connection negotiation failed: client connection to 127.2.164.126:40587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260202 15:06:32.469764 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:32.469517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:51105)
0202 15:06:32.469645 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:32.469708 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260202 15:06:32.470504 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:32.470238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:39489)
0202 15:06:32.470382 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:32.470437 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260202 15:06:32.470705 16054 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260202 15:06:33.161566 15905 heartbeater.cc:499] Master 127.2.164.126:40587 was elected leader, sending a full tablet report...
I20260202 15:06:33.162813 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:33.162239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.65:38977)
0202 15:06:33.162592 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:33.162668 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
W20260202 15:06:33.163156 15905 heartbeater.cc:646] Failed to heartbeat to 127.2.164.126:40587 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
I20260202 15:06:33.471973 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:33.471477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:57473)
0202 15:06:33.471787 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:33.471850 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260202 15:06:34.164404 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:34.163922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.65:36675)
0202 15:06:34.164218 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:34.164280 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260202 15:06:34.473523 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:34.472987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:38849)
0202 15:06:34.473350 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:34.473405 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":125,"threads_started":1}
I20260202 15:06:35.165849 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:35.165371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.65:48777)
0202 15:06:35.165688 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:35.165746 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260202 15:06:35.474802 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:35.474305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:35229)
0202 15:06:35.474641 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:35.474701 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
I20260202 15:06:36.167366 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:36.166896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.65:60991)
0202 15:06:36.167208 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:36.167265 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260202 15:06:36.476083 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:36.475592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:42721)
0202 15:06:36.475918 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:36.475977 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
I20260202 15:06:37.168870 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:37.168376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.65:57541)
0202 15:06:37.168702 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:37.168766 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260202 15:06:37.477559 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0202 15:06:37.477114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40587 (local address 127.2.164.66:49247)
0202 15:06:37.477390 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0202 15:06:37.477458 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
F20260202 15:06:37.514590 2705 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1770044797 (unix time) try "date -d @1770044797" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000a91) received by PID 2705 (TID 0x7f50c13d3d80) from PID 2705; stack trace: ***
@ 0x7f50cd07a980 (unknown) at ??:0
@ 0x7f50ca6b0fb7 gsignal at ??:0
@ 0x7f50ca6b2921 abort at ??:0
@ 0x7f50cb91edcd google::LogMessage::Fail() at ??:0
@ 0x7f50cb922b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f50cb91e7cc google::LogMessage::Flush() at ??:0
@ 0x7f50cb91ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f50ce6abcab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f50ce6a89c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x56402e2d092f kudu::SecurityITest::StartCluster() at ??:0
@ 0x56402e2b8c15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f50cc92f0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f50cc923bdb testing::Test::Run() at ??:0
@ 0x7f50cc923d9d testing::TestInfo::Run() at ??:0
@ 0x7f50cc924377 testing::TestSuite::Run() at ??:0
@ 0x7f50cc92477c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f50cc92f60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f50cc923e63 testing::UnitTest::Run() at ??:0
@ 0x56402e308c7e RUN_ALL_TESTS() at ??:0
@ 0x56402e307e6b main at ??:0
@ 0x7f50ca693bf7 __libc_start_main at ??:0
@ 0x56402e2a006a _start at ??:0
2026-02-02 15:06:37.697 UTC [15636] FATAL: terminating connection due to unexpected postmaster exit
2026-02-02 15:06:37.697 UTC [15533] FATAL: terminating connection due to unexpected postmaster exit
2026-02-02 15:06:37.698 UTC [15532] FATAL: terminating connection due to unexpected postmaster exit
2026-02-02 15:06:37.700 UTC [15530] FATAL: terminating connection due to unexpected postmaster exit
2026-02-02 15:06:37.700 UTC [15529] FATAL: terminating connection due to unexpected postmaster exit
2026-02-02 15:06:37.702 UTC [15528] FATAL: terminating connection due to unexpected postmaster exit