[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:20 dist-test-slave-hss7 krb5kdc[4837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:20 dist-test-slave-hss7 krb5kdc[4837](info): set up 2 sockets
Jan 28 20:50:20 dist-test-slave-hss7 krb5kdc[4837](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260128 20:50:22.407516 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.050s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:22 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:22.582209 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:43181
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:43181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:22.706770 4853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:22.707507 4853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:22.707655 4853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:22.712806 4853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:22.713028 4853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:22.713078 4853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:22.713127 4853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:22.713225 4853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:22.719409 4853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:43181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:43181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:22.720913 4853 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:22.722206 4853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:22.729473 4859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:22.729473 4858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:22.729473 4861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:22.729532 4853 server_base.cc:1060] running on GCE node
I20260128 20:50:22.730675 4853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:22.732218 4853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:22.733513 4853 hybrid_clock.cc:648] HybridClock initialized: now 1769633422733457 us; error 85 us; skew 500 ppm
Jan 28 20:50:22 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633422, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:22.737035 4853 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:22.738514 4853 webserver.cc:492] Webserver started at http://127.4.183.190:32961/ using document root <none> and password file <none>
I20260128 20:50:22.739173 4853 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:22.739262 4853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:22.739691 4853 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:22.741963 4853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "8f18bb1c844047baac7858e3afd24412"
format_stamp: "Formatted at 2026-01-28 20:50:22 on dist-test-slave-hss7"
server_key: "e3f9c35d8c6af29fc9cc1126ec1a43e4"
server_key_iv: "e1a8bc94db65785ff5b32e9bbe496e06"
server_key_version: "encryptionkey@0"
I20260128 20:50:22.742579 4853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "8f18bb1c844047baac7858e3afd24412"
format_stamp: "Formatted at 2026-01-28 20:50:22 on dist-test-slave-hss7"
server_key: "e3f9c35d8c6af29fc9cc1126ec1a43e4"
server_key_iv: "e1a8bc94db65785ff5b32e9bbe496e06"
server_key_version: "encryptionkey@0"
I20260128 20:50:22.746838 4853 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:50:22.750557 4868 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:22.752246 4853 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:50:22.752455 4853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "8f18bb1c844047baac7858e3afd24412"
format_stamp: "Formatted at 2026-01-28 20:50:22 on dist-test-slave-hss7"
server_key: "e3f9c35d8c6af29fc9cc1126ec1a43e4"
server_key_iv: "e1a8bc94db65785ff5b32e9bbe496e06"
server_key_version: "encryptionkey@0"
I20260128 20:50:22.752599 4853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:22.778503 4853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:22.782147 4853 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:22.782409 4853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:22.793860 4853 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:43181
I20260128 20:50:22.793860 4920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:43181 every 8 connection(s)
I20260128 20:50:22.795150 4853 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:22.798739 4921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:22.800854 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 4853
I20260128 20:50:22.801034 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:22.802511 4830 external_mini_cluster.cc:1468] Setting key c9d3e977a640d8b5e3e63b0cc63069ce
I20260128 20:50:22.807489 4921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412: Bootstrap starting.
I20260128 20:50:22.810902 4921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:22.812022 4921 log.cc:826] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:22.814769 4921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412: No bootstrap required, opened a new log
I20260128 20:50:22.819466 4921 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f18bb1c844047baac7858e3afd24412" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43181 } }
I20260128 20:50:22.819931 4921 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:22.820179 4921 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f18bb1c844047baac7858e3afd24412, State: Initialized, Role: FOLLOWER
I20260128 20:50:22.820960 4921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f18bb1c844047baac7858e3afd24412" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43181 } }
Jan 28 20:50:22 dist-test-slave-hss7 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:22.821236 4921 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:22.821336 4921 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:22.821475 4921 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:22.822587 4921 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f18bb1c844047baac7858e3afd24412" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43181 } }
I20260128 20:50:22.823066 4921 leader_election.cc:304] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f18bb1c844047baac7858e3afd24412; no voters:
I20260128 20:50:22.823606 4921 leader_election.cc:290] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:22.824043 4926 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:22.824265 4926 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [term 1 LEADER]: Becoming Leader. State: Replica: 8f18bb1c844047baac7858e3afd24412, State: Running, Role: LEADER
I20260128 20:50:22.824611 4926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f18bb1c844047baac7858e3afd24412" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43181 } }
I20260128 20:50:22.825124 4921 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:22.826524 4928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f18bb1c844047baac7858e3afd24412. Latest consensus state: current_term: 1 leader_uuid: "8f18bb1c844047baac7858e3afd24412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f18bb1c844047baac7858e3afd24412" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43181 } } }
I20260128 20:50:22.827034 4928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:22.827592 4935 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:22.827535 4927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f18bb1c844047baac7858e3afd24412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f18bb1c844047baac7858e3afd24412" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43181 } } }
I20260128 20:50:22.827714 4927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:22.830067 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:22.805739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.4.183.190:43181)
0128 20:50:22.806346 (+ 607us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:22.806371 (+ 25us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:22.806858 (+ 487us) server_negotiation.cc:408] Connection header received
0128 20:50:22.810618 (+ 3760us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:22.810645 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:22.811248 (+ 603us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:22.811721 (+ 473us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:22.814407 (+ 2686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:22.815298 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:22.816252 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:22.816499 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:22.822282 (+ 5783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:22.822311 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:22.822326 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:22.822383 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:22.824819 (+ 2436us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:22.825719 (+ 900us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:22.825727 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:22.825732 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:22.825850 (+ 118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:22.826299 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:22.826302 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:22.826305 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:22.827059 (+ 754us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:22.828590 (+ 1531us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:22.828828 (+ 238us) server_negotiation.cc:300] Negotiation successful
0128 20:50:22.829208 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":156,"threads_started":1}
I20260128 20:50:22.832162 4935 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:22.838848 4935 catalog_manager.cc:1357] Generated new cluster ID: 1cabdd5fe71b43eb80464a2ef31d3342
I20260128 20:50:22.838964 4935 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:22.852156 4935 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:22.853274 4935 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:22.860055 4935 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8f18bb1c844047baac7858e3afd24412: Generated new TSK 0
I20260128 20:50:22.860967 4935 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:22.930044 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43181
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:23.060472 4949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:23.060796 4949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:23.060900 4949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:23.064595 4949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:23.064749 4949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:23.064884 4949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:23.070500 4949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:43181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:23.071910 4949 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:23.072906 4949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:23.080513 4955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:23.080518 4957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:23.080513 4954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:23.081012 4949 server_base.cc:1060] running on GCE node
I20260128 20:50:23.081508 4949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:23.082158 4949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:23.083356 4949 hybrid_clock.cc:648] HybridClock initialized: now 1769633423083322 us; error 49 us; skew 500 ppm
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:23.086760 4949 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:23.088028 4949 webserver.cc:492] Webserver started at http://127.4.183.129:33517/ using document root <none> and password file <none>
I20260128 20:50:23.088660 4949 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:23.088737 4949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:23.088950 4949 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:23.090914 4949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "ad70eeeaf77a474f9c5eb0a5897bacd2"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "45ff5f7e5f9e9ce74fcf6f161f730aa4"
server_key_iv: "671e5187fd3968c1fa548fbb6f624032"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.091472 4949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "ad70eeeaf77a474f9c5eb0a5897bacd2"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "45ff5f7e5f9e9ce74fcf6f161f730aa4"
server_key_iv: "671e5187fd3968c1fa548fbb6f624032"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.095402 4949 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260128 20:50:23.098205 4964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.099509 4949 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260128 20:50:23.099691 4949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "ad70eeeaf77a474f9c5eb0a5897bacd2"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "45ff5f7e5f9e9ce74fcf6f161f730aa4"
server_key_iv: "671e5187fd3968c1fa548fbb6f624032"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.099824 4949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:23.126784 4949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:23.130796 4949 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:23.131001 4949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:23.131639 4949 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:23.132711 4949 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:23.132768 4949 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.132849 4949 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:23.132880 4949 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.144308 4949 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:43917
I20260128 20:50:23.144323 5077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:43917 every 8 connection(s)
I20260128 20:50:23.145545 4949 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:50:23.148336 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 4949
I20260128 20:50:23.148475 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:23.148717 4830 external_mini_cluster.cc:1468] Setting key 6fd5755475b4b6cd65e5453c3559208e
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:50:23.162734 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.147798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:35539 (local address 127.4.183.190:43181)
0128 20:50:23.147985 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:23.147989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:23.148791 (+ 802us) server_negotiation.cc:408] Connection header received
0128 20:50:23.149992 (+ 1201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:23.149995 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:23.150105 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:23.150218 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:23.152176 (+ 1958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.152720 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.153557 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.153759 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.156655 (+ 2896us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:23.156682 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:23.156685 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:23.156720 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:23.158793 (+ 2073us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.160129 (+ 1336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.160136 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.160138 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.160254 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.160705 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.160709 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.160711 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.161037 (+ 326us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:23.161378 (+ 341us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:23.162261 (+ 883us) server_negotiation.cc:300] Negotiation successful
0128 20:50:23.162544 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260128 20:50:23.163380 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.148082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43181 (local address 127.4.183.129:35539)
0128 20:50:23.148604 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:23.148650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:23.149755 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:23.150447 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:23.150463 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:23.150968 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:23.151986 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:23.152003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.152879 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.152883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:23.153407 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:23.153417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.153712 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.154672 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:23.154708 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:23.156410 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:23.159081 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:23.159092 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:23.159120 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:23.159892 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:23.160392 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:23.160396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:23.160398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:23.160585 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:23.161379 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:23.161392 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:23.161807 (+ 415us) client_negotiation.cc:769] Sending connection context
0128 20:50:23.162194 (+ 387us) client_negotiation.cc:240] Negotiation successful
0128 20:50:23.162544 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":180,"threads_started":1}
I20260128 20:50:23.165028 5078 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43181
I20260128 20:50:23.165446 5078 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:23.166155 5078 heartbeater.cc:507] Master 127.4.183.190:43181 requested a full tablet report, sending...
I20260128 20:50:23.168011 4885 ts_manager.cc:194] Registered new tserver with Master: ad70eeeaf77a474f9c5eb0a5897bacd2 (127.4.183.129:43917)
I20260128 20:50:23.169425 4885 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:35539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:23.209637 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43181
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:23.335862 5085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:23.336308 5085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:23.336444 5085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:23.341795 5085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:23.341921 5085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:23.342095 5085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:23.347539 5085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:43181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:23.348922 5085 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:23.350003 5085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:23.357935 5093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:23.357921 5090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:23.357921 5091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:23.358253 5085 server_base.cc:1060] running on GCE node
I20260128 20:50:23.359079 5085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:23.360039 5085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:23.361299 5085 hybrid_clock.cc:648] HybridClock initialized: now 1769633423361213 us; error 120 us; skew 500 ppm
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:23.365006 5085 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:23.366540 5085 webserver.cc:492] Webserver started at http://127.4.183.130:43883/ using document root <none> and password file <none>
I20260128 20:50:23.367162 5085 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:23.367229 5085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:23.367429 5085 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:23.369550 5085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "0320d09baf474b298affd9f324e6b3f5"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "992d72a752dbd543cc1359755e101bc2"
server_key_iv: "f10dc8644e802b1e79e477b85802923c"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.370085 5085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "0320d09baf474b298affd9f324e6b3f5"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "992d72a752dbd543cc1359755e101bc2"
server_key_iv: "f10dc8644e802b1e79e477b85802923c"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.374497 5085 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:23.377803 5100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.379397 5085 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:50:23.379626 5085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "0320d09baf474b298affd9f324e6b3f5"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "992d72a752dbd543cc1359755e101bc2"
server_key_iv: "f10dc8644e802b1e79e477b85802923c"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.379769 5085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:23.392405 5085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:23.395906 5085 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:23.396178 5085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:23.396956 5085 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:23.398016 5085 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:23.398103 5085 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.398182 5085 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:23.398229 5085 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.409336 5085 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:45607
I20260128 20:50:23.409353 5213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:45607 every 8 connection(s)
I20260128 20:50:23.410482 5085 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:23.417416 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5085
I20260128 20:50:23.417541 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:23.417752 4830 external_mini_cluster.cc:1468] Setting key b307588d78f1ff69e639735f743a31e8
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:23.426368 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.412665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:41275 (local address 127.4.183.190:43181)
0128 20:50:23.412876 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:23.412882 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:23.413669 (+ 787us) server_negotiation.cc:408] Connection header received
0128 20:50:23.414749 (+ 1080us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:23.414754 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:23.414856 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:23.415170 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:23.417250 (+ 2080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.417913 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.418763 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.418925 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.421682 (+ 2757us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:23.421707 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:23.421710 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:23.421746 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:23.423715 (+ 1969us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.424455 (+ 740us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.424462 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.424466 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.424535 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.424924 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.424930 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.424935 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.425208 (+ 273us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:23.425321 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:23.425970 (+ 649us) server_negotiation.cc:300] Negotiation successful
0128 20:50:23.426141 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260128 20:50:23.427022 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.412906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43181 (local address 127.4.183.130:41275)
0128 20:50:23.413483 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:23.413535 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:23.414363 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:23.415367 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:23.415379 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:23.415848 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:23.416908 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:23.416928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.418089 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.418093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:23.418603 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:23.418616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.418915 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.419688 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:23.419720 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:23.421473 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:23.423936 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:23.423952 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:23.423965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:23.424312 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:23.424672 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:23.424675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:23.424677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:23.424806 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:23.425386 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:23.425393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:23.425676 (+ 283us) client_negotiation.cc:769] Sending connection context
0128 20:50:23.425884 (+ 208us) client_negotiation.cc:240] Negotiation successful
0128 20:50:23.426149 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":172,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:50:23.428617 5214 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43181
I20260128 20:50:23.428947 5214 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:23.429643 5214 heartbeater.cc:507] Master 127.4.183.190:43181 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:50:23.430961 4885 ts_manager.cc:194] Registered new tserver with Master: 0320d09baf474b298affd9f324e6b3f5 (127.4.183.130:45607)
I20260128 20:50:23.431600 4885 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:41275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:23.476615 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43181
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:23.600091 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:23.600423 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:23.600522 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:23.604189 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:23.604331 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:23.604454 5221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:23.609705 5221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:43181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:23.610984 5221 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:23.611928 5221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:23.619181 5226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:23.619124 5229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:23.619369 5221 server_base.cc:1060] running on GCE node
W20260128 20:50:23.619124 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:23.619961 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:23.620632 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:23.621824 5221 hybrid_clock.cc:648] HybridClock initialized: now 1769633423621791 us; error 52 us; skew 500 ppm
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:23.625881 5221 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:23.627439 5221 webserver.cc:492] Webserver started at http://127.4.183.131:32937/ using document root <none> and password file <none>
I20260128 20:50:23.628089 5221 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:23.628182 5221 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:23.628427 5221 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:23.630532 5221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "06759f141f0e405b83d4895dc9a0e463"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "62d9d1b20dd41192c9a17555d4010f85"
server_key_iv: "cbb7f91d53a8583723e6fcd0c51fb846"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.631148 5221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "06759f141f0e405b83d4895dc9a0e463"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "62d9d1b20dd41192c9a17555d4010f85"
server_key_iv: "cbb7f91d53a8583723e6fcd0c51fb846"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.635335 5221 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:50:23.638217 5236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.639652 5221 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:50:23.639865 5221 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "06759f141f0e405b83d4895dc9a0e463"
format_stamp: "Formatted at 2026-01-28 20:50:23 on dist-test-slave-hss7"
server_key: "62d9d1b20dd41192c9a17555d4010f85"
server_key_iv: "cbb7f91d53a8583723e6fcd0c51fb846"
server_key_version: "encryptionkey@0"
I20260128 20:50:23.640031 5221 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:23.658222 5221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:23.662091 5221 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:23.662346 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:23.662989 5221 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:23.664021 5221 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:23.664103 5221 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.664191 5221 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:23.664243 5221 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:23.674613 5221 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:36391
I20260128 20:50:23.674650 5349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:36391 every 8 connection(s)
I20260128 20:50:23.675722 5221 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:23.684880 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5221
I20260128 20:50:23.684999 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:23.685340 4830 external_mini_cluster.cc:1468] Setting key 48f3fb9827fe3bb8e38b5f7ffe2b25af
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:23.695451 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.677976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:51117 (local address 127.4.183.190:43181)
0128 20:50:23.678161 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:23.678166 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:23.679169 (+ 1003us) server_negotiation.cc:408] Connection header received
0128 20:50:23.680382 (+ 1213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:23.680385 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:23.680445 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:23.680535 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:23.682620 (+ 2085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.683216 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.684017 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.684198 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.690618 (+ 6420us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:23.690642 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:23.690645 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:23.690681 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:23.692281 (+ 1600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.692968 (+ 687us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.692972 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.692973 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.693027 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.693559 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.693562 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.693564 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.693765 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:23.693854 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:23.695080 (+ 1226us) server_negotiation.cc:300] Negotiation successful
0128 20:50:23.695298 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260128 20:50:23.696532 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.678355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43181 (local address 127.4.183.131:51117)
0128 20:50:23.678946 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:23.679004 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:23.680045 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:23.680754 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:23.680766 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:23.681361 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:23.682381 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:23.682398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.683420 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.683423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:23.683874 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:23.683882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.684112 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.688000 (+ 3888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:23.688047 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:23.690383 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:23.692457 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:23.692463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:23.692476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:23.692832 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:23.693273 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:23.693277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:23.693278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:23.693433 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:23.694090 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:23.694104 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:23.694563 (+ 459us) client_negotiation.cc:769] Sending connection context
0128 20:50:23.694874 (+ 311us) client_negotiation.cc:240] Negotiation successful
0128 20:50:23.695398 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":165,"threads_started":1}
I20260128 20:50:23.698606 5350 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43181
I20260128 20:50:23.698987 5350 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:23.699816 5350 heartbeater.cc:507] Master 127.4.183.190:43181 requested a full tablet report, sending...
I20260128 20:50:23.701035 4885 ts_manager.cc:194] Registered new tserver with Master: 06759f141f0e405b83d4895dc9a0e463 (127.4.183.131:36391)
I20260128 20:50:23.701732 4885 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:51117
I20260128 20:50:23.713812 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:23.743202 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.731487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.4.183.129:43917)
0128 20:50:23.731802 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:23.731812 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:23.731939 (+ 127us) server_negotiation.cc:408] Connection header received
0128 20:50:23.732154 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:23.732165 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:23.732449 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:23.732637 (+ 188us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:23.733608 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.734359 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.735504 (+ 1145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.735726 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.738700 (+ 2974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:23.738756 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:23.738769 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:23.738809 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:23.740928 (+ 2119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.741702 (+ 774us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.741708 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.741710 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.741788 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.742070 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.742073 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.742075 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.742338 (+ 263us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:23.742520 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:23.742848 (+ 328us) server_negotiation.cc:300] Negotiation successful
0128 20:50:23.743014 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
W20260128 20:50:23.744414 4992 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:23.774184 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:23.763462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.4.183.129:43917)
0128 20:50:23.763696 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:23.763704 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:23.763729 (+ 25us) server_negotiation.cc:408] Connection header received
0128 20:50:23.763967 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:23.763970 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:23.764044 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:23.764153 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:23.765449 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.766258 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:23.767350 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:23.767593 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:23.770409 (+ 2816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:23.770435 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:23.770438 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:23.770482 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:23.772319 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.772867 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.772871 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.772872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.772929 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:23.773238 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:23.773243 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:23.773245 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:23.773428 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:23.773589 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:23.773832 (+ 243us) server_negotiation.cc:300] Negotiation successful
0128 20:50:23.773988 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260128 20:50:23.775434 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 4949
I20260128 20:50:23.782859 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5085
I20260128 20:50:23.791309 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5221
I20260128 20:50:23.797863 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 4853
2026-01-28T20:50:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3473 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[5364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[5364](info): set up 2 sockets
Jan 28 20:50:23 dist-test-slave-hss7 krb5kdc[5364](info): commencing operation
krb5kdc: starting...
W20260128 20:50:25.875537 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.053s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:25 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:26.050228 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:38509
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:26.185144 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:26.185583 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:26.185709 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:26.190084 5380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:26.190268 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:26.190313 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:26.190346 5380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:26.190389 5380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:26.197549 5380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:26.199056 5380 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:26.200811 5380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:26.210254 5386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:26.210254 5385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:26.210554 5380 server_base.cc:1060] running on GCE node
W20260128 20:50:26.210254 5388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:26.211436 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:26.212638 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:26.214052 5380 hybrid_clock.cc:648] HybridClock initialized: now 1769633426213970 us; error 91 us; skew 500 ppm
Jan 28 20:50:26 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:26.218158 5380 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:26.220161 5380 webserver.cc:492] Webserver started at http://127.4.183.190:37421/ using document root <none> and password file <none>
I20260128 20:50:26.221292 5380 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:26.221427 5380 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:26.221688 5380 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:26.223855 5380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "5f3b531a780933a8dd5f7d38c31771a6"
server_key_iv: "f2fc7228b28e8ecb518bcddef155a6dd"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.224408 5380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "5f3b531a780933a8dd5f7d38c31771a6"
server_key_iv: "f2fc7228b28e8ecb518bcddef155a6dd"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.228930 5380 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:50:26.232015 5395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.234113 5380 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:26.234457 5380 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "5f3b531a780933a8dd5f7d38c31771a6"
server_key_iv: "f2fc7228b28e8ecb518bcddef155a6dd"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.234715 5380 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:26.257431 5380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:26.267584 5380 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:26.267905 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:26.278875 5380 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:38509
I20260128 20:50:26.278923 5447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:38509 every 8 connection(s)
I20260128 20:50:26.280144 5380 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:26.283525 5448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:26.290041 5448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c: Bootstrap starting.
I20260128 20:50:26.290359 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5380
I20260128 20:50:26.290529 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:26.290823 4830 external_mini_cluster.cc:1468] Setting key 7511793052231982f7755712e93d5b8c
I20260128 20:50:26.293316 5448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:26.294190 5448 log.cc:826] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:26.296692 5448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c: No bootstrap required, opened a new log
Jan 28 20:50:26 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:26.299922 5448 raft_consensus.cc:359] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38509 } }
I20260128 20:50:26.300199 5448 raft_consensus.cc:385] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:26.300280 5448 raft_consensus.cc:740] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b706dcb9c948e28f8ebb7a95cb3b0c, State: Initialized, Role: FOLLOWER
I20260128 20:50:26.300885 5448 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38509 } }
I20260128 20:50:26.301064 5448 raft_consensus.cc:399] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:26.301314 5448 raft_consensus.cc:493] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:26.301467 5448 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:26.302834 5448 raft_consensus.cc:515] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38509 } }
I20260128 20:50:26.303332 5448 leader_election.cc:304] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b706dcb9c948e28f8ebb7a95cb3b0c; no voters:
I20260128 20:50:26.303812 5448 leader_election.cc:290] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:26.304350 5453 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:26.304611 5453 raft_consensus.cc:697] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [term 1 LEADER]: Becoming Leader. State: Replica: d1b706dcb9c948e28f8ebb7a95cb3b0c, State: Running, Role: LEADER
I20260128 20:50:26.304965 5453 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38509 } }
I20260128 20:50:26.305287 5448 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:26.306620 5455 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b706dcb9c948e28f8ebb7a95cb3b0c. Latest consensus state: current_term: 1 leader_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38509 } } }
I20260128 20:50:26.306787 5455 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:26.307165 5454 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b706dcb9c948e28f8ebb7a95cb3b0c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38509 } } }
I20260128 20:50:26.307216 5462 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:26.307282 5454 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:26.307983 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:26.292603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.4.183.190:38509)
0128 20:50:26.293321 (+ 718us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:26.293338 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:26.293445 (+ 107us) server_negotiation.cc:408] Connection header received
0128 20:50:26.294197 (+ 752us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:26.294219 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:26.294625 (+ 406us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:26.294961 (+ 336us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:26.295934 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.297056 (+ 1122us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:26.297876 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.298178 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:26.300594 (+ 2416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:26.300624 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:26.300639 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:26.300700 (+ 61us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:26.303141 (+ 2441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:26.303745 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:26.303755 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:26.303764 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:26.303878 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:26.304219 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:26.304222 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:26.304224 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:26.304701 (+ 477us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:26.305726 (+ 1025us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:26.305929 (+ 203us) server_negotiation.cc:300] Negotiation successful
0128 20:50:26.306281 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":218,"threads_started":1}
I20260128 20:50:26.311626 5462 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:26.318832 5462 catalog_manager.cc:1357] Generated new cluster ID: e07c32eef0164dafb02c26e1cb5bf528
I20260128 20:50:26.318939 5462 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:26.338900 5462 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:26.340811 5462 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:26.369195 5462 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d1b706dcb9c948e28f8ebb7a95cb3b0c: Generated new TSK 0
I20260128 20:50:26.370149 5462 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:26.437655 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:38509
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:26.581760 5476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:26.582096 5476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:26.582223 5476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:26.586016 5476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:26.586153 5476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:26.586278 5476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:26.592219 5476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:38509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:26.594036 5476 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:26.595089 5476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:26.602960 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:26.603026 5476 server_base.cc:1060] running on GCE node
W20260128 20:50:26.602960 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:26.602960 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:26.603771 5476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:26.604508 5476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:26.605733 5476 hybrid_clock.cc:648] HybridClock initialized: now 1769633426605694 us; error 60 us; skew 500 ppm
Jan 28 20:50:26 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:26.610239 5476 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:26.612190 5476 webserver.cc:492] Webserver started at http://127.4.183.129:44187/ using document root <none> and password file <none>
I20260128 20:50:26.613013 5476 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:26.613131 5476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:26.613459 5476 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:26.615566 5476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "6673ba6fdde845608628fadd4637f7dd"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "790a313b4d97975b1f28371b5b0a71bc"
server_key_iv: "fd4793acaedf47f1a1edfa19850d74c0"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.616238 5476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "6673ba6fdde845608628fadd4637f7dd"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "790a313b4d97975b1f28371b5b0a71bc"
server_key_iv: "fd4793acaedf47f1a1edfa19850d74c0"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.620563 5476 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:50:26.623948 5491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.625872 5476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:26.626080 5476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "6673ba6fdde845608628fadd4637f7dd"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "790a313b4d97975b1f28371b5b0a71bc"
server_key_iv: "fd4793acaedf47f1a1edfa19850d74c0"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.626255 5476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:26.652691 5476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:26.657469 5476 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:26.657768 5476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:26.658542 5476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:26.659612 5476 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:26.659693 5476 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.659776 5476 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:26.659819 5476 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.671921 5476 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:33093
I20260128 20:50:26.671936 5604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:33093 every 8 connection(s)
I20260128 20:50:26.673416 5476 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:50:26.677646 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5476
I20260128 20:50:26.677803 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:26.678133 4830 external_mini_cluster.cc:1468] Setting key 53201b1167bdbd7135021d3171205b96
Jan 28 20:50:26 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
I20260128 20:50:26.691347 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:26.675548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:39545 (local address 127.4.183.190:38509)
0128 20:50:26.675745 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:26.675750 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:26.676652 (+ 902us) server_negotiation.cc:408] Connection header received
0128 20:50:26.678124 (+ 1472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:26.678129 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:26.678245 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:26.678375 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:26.680620 (+ 2245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.681228 (+ 608us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:26.681962 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.682190 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:26.684889 (+ 2699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:26.684928 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:26.684935 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:26.684982 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:26.687156 (+ 2174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:26.688394 (+ 1238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:26.688398 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:26.688400 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:26.688472 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:26.689103 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:26.689127 (+ 24us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:26.689129 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:26.689422 (+ 293us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:26.689527 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:26.690796 (+ 1269us) server_negotiation.cc:300] Negotiation successful
0128 20:50:26.691073 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260128 20:50:26.692523 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:26.675851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:38509 (local address 127.4.183.129:39545)
0128 20:50:26.676448 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:26.676493 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:26.677628 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:26.678623 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:26.678634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:26.679232 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:26.680429 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:26.680450 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.681390 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:26.681394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:26.681820 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:26.681830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.682124 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:26.682893 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:26.682927 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:26.684644 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:26.687483 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:26.687495 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:26.687515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:26.688192 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:26.688699 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:26.688704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:26.688706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:26.688884 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:26.689667 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:26.689698 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:26.690334 (+ 636us) client_negotiation.cc:769] Sending connection context
0128 20:50:26.690642 (+ 308us) client_negotiation.cc:240] Negotiation successful
0128 20:50:26.691047 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:50:26.694849 5605 heartbeater.cc:344] Connected to a master server at 127.4.183.190:38509
I20260128 20:50:26.695422 5605 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:26.696413 5605 heartbeater.cc:507] Master 127.4.183.190:38509 requested a full tablet report, sending...
I20260128 20:50:26.699090 5412 ts_manager.cc:194] Registered new tserver with Master: 6673ba6fdde845608628fadd4637f7dd (127.4.183.129:33093)
I20260128 20:50:26.700520 5412 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:39545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:26.746027 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:38509
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:26.880800 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:26.881095 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:26.881253 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:26.885205 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:26.885300 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:26.885391 5612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:26.890219 5612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:38509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:26.891507 5612 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:26.892481 5612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:26.900260 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:26.900481 5612 server_base.cc:1060] running on GCE node
W20260128 20:50:26.900260 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:26.900382 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:26.901026 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:26.901796 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:26.902983 5612 hybrid_clock.cc:648] HybridClock initialized: now 1769633426902956 us; error 45 us; skew 500 ppm
Jan 28 20:50:26 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:26.906342 5612 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:26.909315 5612 webserver.cc:492] Webserver started at http://127.4.183.130:43713/ using document root <none> and password file <none>
I20260128 20:50:26.910040 5612 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:26.910116 5612 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:26.910389 5612 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:26.912436 5612 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "6d349d30c30b4c82b17fdf98a576cd47"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "ac015104f2ea5904e9c481a43eed26f4"
server_key_iv: "2a26b2832a3c129001a15cfdfed0c786"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.913039 5612 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "6d349d30c30b4c82b17fdf98a576cd47"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "ac015104f2ea5904e9c481a43eed26f4"
server_key_iv: "2a26b2832a3c129001a15cfdfed0c786"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.917491 5612 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:26.920260 5627 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.921692 5612 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:26.921859 5612 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "6d349d30c30b4c82b17fdf98a576cd47"
format_stamp: "Formatted at 2026-01-28 20:50:26 on dist-test-slave-hss7"
server_key: "ac015104f2ea5904e9c481a43eed26f4"
server_key_iv: "2a26b2832a3c129001a15cfdfed0c786"
server_key_version: "encryptionkey@0"
I20260128 20:50:26.921989 5612 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:26.938903 5612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:26.942586 5612 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:26.942850 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:26.943495 5612 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:26.944553 5612 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:26.944625 5612 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.944695 5612 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:26.944746 5612 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:26.957952 5612 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36785
I20260128 20:50:26.957965 5740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36785 every 8 connection(s)
I20260128 20:50:26.959084 5612 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:26.965468 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5612
I20260128 20:50:26.965633 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:26.965893 4830 external_mini_cluster.cc:1468] Setting key 862b7b2ed8c0732ec3eeab8e14c70cde
Jan 28 20:50:26 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:26.976317 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:26.961434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:55715 (local address 127.4.183.190:38509)
0128 20:50:26.961663 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:26.961668 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:26.962385 (+ 717us) server_negotiation.cc:408] Connection header received
0128 20:50:26.963401 (+ 1016us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:26.963406 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:26.963475 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:26.963592 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:26.965585 (+ 1993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.966238 (+ 653us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:26.966999 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.967203 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:26.971122 (+ 3919us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:26.971163 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:26.971303 (+ 140us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:26.971367 (+ 64us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:26.973477 (+ 2110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:26.974188 (+ 711us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:26.974196 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:26.974199 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:26.974267 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:26.974604 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:26.974610 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:26.974616 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:26.974875 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:26.975001 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:26.975927 (+ 926us) server_negotiation.cc:300] Negotiation successful
0128 20:50:26.976104 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260128 20:50:26.976992 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:26.961598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:38509 (local address 127.4.183.130:55715)
0128 20:50:26.962177 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:26.962222 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:26.963129 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:26.963793 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:26.963803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:26.964270 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:26.965361 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:26.965381 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.966397 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:26.966401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:26.966835 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:26.966844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:26.967200 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:26.968196 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:26.968437 (+ 241us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:26.970821 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:26.973690 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:26.973698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:26.973712 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:26.974037 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:26.974379 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:26.974381 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:26.974383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:26.974483 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:26.975024 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:26.975032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:26.975365 (+ 333us) client_negotiation.cc:769] Sending connection context
0128 20:50:26.975824 (+ 459us) client_negotiation.cc:240] Negotiation successful
0128 20:50:26.976105 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:50:26.978535 5741 heartbeater.cc:344] Connected to a master server at 127.4.183.190:38509
I20260128 20:50:26.978878 5741 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:26.979547 5741 heartbeater.cc:507] Master 127.4.183.190:38509 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:50:26.980907 5412 ts_manager.cc:194] Registered new tserver with Master: 6d349d30c30b4c82b17fdf98a576cd47 (127.4.183.130:36785)
I20260128 20:50:26.981761 5412 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:55715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:27.024984 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:38509
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:27.148409 5748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:27.148694 5748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:27.148757 5748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:27.152395 5748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:27.152489 5748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:27.152580 5748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:27.157546 5748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:38509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:27.158805 5748 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:27.159763 5748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:27.167791 5754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:27.167862 5748 server_base.cc:1060] running on GCE node
W20260128 20:50:27.167791 5756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:27.167809 5753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:27.169025 5748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:27.169937 5748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:27.171198 5748 hybrid_clock.cc:648] HybridClock initialized: now 1769633427171139 us; error 98 us; skew 500 ppm
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633427, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:27.175217 5748 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:27.176779 5748 webserver.cc:492] Webserver started at http://127.4.183.131:37609/ using document root <none> and password file <none>
I20260128 20:50:27.177560 5748 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:27.177624 5748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:27.177853 5748 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:27.179816 5748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "8eb4ae828a934d81b9b4bd485907d4d1"
format_stamp: "Formatted at 2026-01-28 20:50:27 on dist-test-slave-hss7"
server_key: "6395c38b25de3f0f26b874a2c09a492c"
server_key_iv: "753aef90d391d3c346ab866a54cb99ad"
server_key_version: "encryptionkey@0"
I20260128 20:50:27.180402 5748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "8eb4ae828a934d81b9b4bd485907d4d1"
format_stamp: "Formatted at 2026-01-28 20:50:27 on dist-test-slave-hss7"
server_key: "6395c38b25de3f0f26b874a2c09a492c"
server_key_iv: "753aef90d391d3c346ab866a54cb99ad"
server_key_version: "encryptionkey@0"
I20260128 20:50:27.184509 5748 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260128 20:50:27.187731 5763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:27.189805 5748 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260128 20:50:27.190023 5748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "8eb4ae828a934d81b9b4bd485907d4d1"
format_stamp: "Formatted at 2026-01-28 20:50:27 on dist-test-slave-hss7"
server_key: "6395c38b25de3f0f26b874a2c09a492c"
server_key_iv: "753aef90d391d3c346ab866a54cb99ad"
server_key_version: "encryptionkey@0"
I20260128 20:50:27.190182 5748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:27.215183 5748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:27.218724 5748 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:27.218984 5748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:27.219640 5748 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:27.220745 5748 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:27.220825 5748 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:27.220922 5748 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:27.220974 5748 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:27.235649 5748 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:37585
I20260128 20:50:27.235783 5876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:37585 every 8 connection(s)
I20260128 20:50:27.237293 5748 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:27.243232 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5748
I20260128 20:50:27.243366 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:27.243652 4830 external_mini_cluster.cc:1468] Setting key 49bfe9a10ff415250c925e88eab06306
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633427, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:27.252795 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.239646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:50167 (local address 127.4.183.190:38509)
0128 20:50:27.239794 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.239798 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.240653 (+ 855us) server_negotiation.cc:408] Connection header received
0128 20:50:27.242059 (+ 1406us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.242062 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.242134 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.242220 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:27.244522 (+ 2302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.245276 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.246255 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.246445 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.249046 (+ 2601us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:27.249068 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:27.249070 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:27.249101 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:27.250663 (+ 1562us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.251232 (+ 569us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.251235 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.251237 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.251289 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.251605 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.251608 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.251611 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.251787 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:27.251892 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.252467 (+ 575us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.252611 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260128 20:50:27.253544 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.239869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:38509 (local address 127.4.183.131:50167)
0128 20:50:27.240435 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:27.240478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:27.241752 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:27.242479 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:27.242490 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:27.242960 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:27.244261 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.244287 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.245632 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.245636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:27.246118 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.246129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.246438 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.247145 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:27.247177 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:27.248848 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:27.250819 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:27.250827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:27.250842 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:27.251119 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:27.251405 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:27.251407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:27.251409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:27.251508 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:27.251903 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:27.251912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:27.252189 (+ 277us) client_negotiation.cc:769] Sending connection context
0128 20:50:27.252434 (+ 245us) client_negotiation.cc:240] Negotiation successful
0128 20:50:27.252730 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":140,"threads_started":1}
I20260128 20:50:27.255074 5877 heartbeater.cc:344] Connected to a master server at 127.4.183.190:38509
I20260128 20:50:27.255532 5877 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:27.256520 5877 heartbeater.cc:507] Master 127.4.183.190:38509 requested a full tablet report, sending...
I20260128 20:50:27.257967 5411 ts_manager.cc:194] Registered new tserver with Master: 8eb4ae828a934d81b9b4bd485907d4d1 (127.4.183.131:37585)
I20260128 20:50:27.258988 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 20:50:27.258970 5411 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:50167
I20260128 20:50:27.271622 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.262637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.4.183.190:38509)
0128 20:50:27.262824 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.262829 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.262945 (+ 116us) server_negotiation.cc:408] Connection header received
0128 20:50:27.263151 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.263154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.263206 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.263308 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:27.264222 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.264736 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.265534 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.265741 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.266938 (+ 1197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:27.266968 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:27.266971 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:27.266999 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:27.268982 (+ 1983us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.269806 (+ 824us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.269811 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.269813 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.269890 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.270198 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.270202 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.270204 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.270634 (+ 430us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:27.270877 (+ 243us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.271200 (+ 323us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.271403 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260128 20:50:27.277518 5411 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:50:27.280231 5411 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:27.296332 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.290881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:33093 (local address 127.0.0.1:44976)
0128 20:50:27.291755 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:27.291800 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:27.292029 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:27.292785 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:27.292791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:27.292811 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:27.293163 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.293173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.294745 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.294749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:27.295588 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.295598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.295814 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.295891 (+ 77us) client_negotiation.cc:769] Sending connection context
0128 20:50:27.296005 (+ 114us) client_negotiation.cc:240] Negotiation successful
0128 20:50:27.296135 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":134,"threads_started":1}
I20260128 20:50:27.296875 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.291050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.4.183.129:33093)
0128 20:50:27.291483 (+ 433us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.291493 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.291889 (+ 396us) server_negotiation.cc:408] Connection header received
0128 20:50:27.292229 (+ 340us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.292235 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.292459 (+ 224us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.292575 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:27.293318 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.294574 (+ 1256us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.295769 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.296381 (+ 612us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.296539 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.296619 (+ 80us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.296731 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":76,"threads_started":1}
I20260128 20:50:27.296908 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.291690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:36785 (local address 127.0.0.1:49832)
0128 20:50:27.292257 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:27.292271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:27.292341 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:27.293403 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:27.293407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:27.293446 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:27.293690 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.293698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.295243 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.295247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:27.296335 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.296361 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.296560 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.296589 (+ 29us) client_negotiation.cc:769] Sending connection context
0128 20:50:27.296639 (+ 50us) client_negotiation.cc:240] Negotiation successful
0128 20:50:27.296721 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260128 20:50:27.297825 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:37585 (local address 127.0.0.1:55882)
0128 20:50:27.292560 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:27.292573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:27.292639 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:27.293392 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:27.293395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:27.293413 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:27.293694 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.293701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.294959 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.294963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:27.296335 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.296361 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.297507 (+ 1146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.297534 (+ 27us) client_negotiation.cc:769] Sending connection context
0128 20:50:27.297593 (+ 59us) client_negotiation.cc:240] Negotiation successful
0128 20:50:27.297678 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"spinlock_wait_cycles":896,"thread_start_us":86,"threads_started":1}
I20260128 20:50:27.297879 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.292191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.4.183.130:36785)
0128 20:50:27.292899 (+ 708us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.292905 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.292927 (+ 22us) server_negotiation.cc:408] Connection header received
0128 20:50:27.293008 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.293014 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.293233 (+ 219us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.293835 (+ 602us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:27.293982 (+ 147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.295064 (+ 1082us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.296789 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.297467 (+ 678us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.297583 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.297648 (+ 65us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.297739 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":137,"threads_started":1}
I20260128 20:50:27.299506 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.292417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.4.183.131:37585)
0128 20:50:27.292785 (+ 368us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.292792 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.292815 (+ 23us) server_negotiation.cc:408] Connection header received
0128 20:50:27.292893 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.292898 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.293064 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.293239 (+ 175us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:27.293830 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.294828 (+ 998us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.298289 (+ 3461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.299013 (+ 724us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.299186 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.299265 (+ 79us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.299381 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":141,"threads_started":1}
I20260128 20:50:27.300439 5675 tablet_service.cc:1505] Processing CreateTablet for tablet 30008a07a9864c9fbe5b5571dea01e48 (DEFAULT_TABLE table=test-table [id=eb343728414349f693bfd35143ca7dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:27.300439 5539 tablet_service.cc:1505] Processing CreateTablet for tablet 30008a07a9864c9fbe5b5571dea01e48 (DEFAULT_TABLE table=test-table [id=eb343728414349f693bfd35143ca7dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:27.301606 5675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30008a07a9864c9fbe5b5571dea01e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:27.301606 5539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30008a07a9864c9fbe5b5571dea01e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:27.302491 5803 tablet_service.cc:1505] Processing CreateTablet for tablet 30008a07a9864c9fbe5b5571dea01e48 (DEFAULT_TABLE table=test-table [id=eb343728414349f693bfd35143ca7dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:27.303552 5803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30008a07a9864c9fbe5b5571dea01e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:27.309803 5895 tablet_bootstrap.cc:492] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: Bootstrap starting.
I20260128 20:50:27.310053 5894 tablet_bootstrap.cc:492] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: Bootstrap starting.
I20260128 20:50:27.310721 5896 tablet_bootstrap.cc:492] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: Bootstrap starting.
I20260128 20:50:27.312461 5895 tablet_bootstrap.cc:654] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:27.312541 5894 tablet_bootstrap.cc:654] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:27.313022 5896 tablet_bootstrap.cc:654] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:27.313424 5895 log.cc:826] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:27.313442 5894 log.cc:826] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:27.313907 5896 log.cc:826] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:27.315816 5895 tablet_bootstrap.cc:492] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: No bootstrap required, opened a new log
I20260128 20:50:27.315816 5894 tablet_bootstrap.cc:492] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: No bootstrap required, opened a new log
I20260128 20:50:27.315816 5896 tablet_bootstrap.cc:492] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: No bootstrap required, opened a new log
I20260128 20:50:27.316105 5895 ts_tablet_manager.cc:1403] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:50:27.316105 5896 ts_tablet_manager.cc:1403] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:50:27.316105 5894 ts_tablet_manager.cc:1403] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:50:27.319491 5895 raft_consensus.cc:359] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.319619 5894 raft_consensus.cc:359] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.319792 5895 raft_consensus.cc:385] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:27.319871 5895 raft_consensus.cc:740] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d349d30c30b4c82b17fdf98a576cd47, State: Initialized, Role: FOLLOWER
I20260128 20:50:27.319914 5894 raft_consensus.cc:385] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:27.320026 5894 raft_consensus.cc:740] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6673ba6fdde845608628fadd4637f7dd, State: Initialized, Role: FOLLOWER
I20260128 20:50:27.319942 5896 raft_consensus.cc:359] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.320178 5896 raft_consensus.cc:385] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:27.320235 5896 raft_consensus.cc:740] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb4ae828a934d81b9b4bd485907d4d1, State: Initialized, Role: FOLLOWER
I20260128 20:50:27.320591 5894 consensus_queue.cc:260] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.320602 5895 consensus_queue.cc:260] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.320647 5896 consensus_queue.cc:260] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.321710 5894 ts_tablet_manager.cc:1434] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260128 20:50:27.321710 5896 ts_tablet_manager.cc:1434] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260128 20:50:27.321735 5895 ts_tablet_manager.cc:1434] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260128 20:50:27.321764 5605 heartbeater.cc:499] Master 127.4.183.190:38509 was elected leader, sending a full tablet report...
I20260128 20:50:27.322093 5741 heartbeater.cc:499] Master 127.4.183.190:38509 was elected leader, sending a full tablet report...
I20260128 20:50:27.322268 5877 heartbeater.cc:499] Master 127.4.183.190:38509 was elected leader, sending a full tablet report...
W20260128 20:50:27.425119 5606 tablet.cc:2378] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 20:50:27.460441 5742 tablet.cc:2378] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 20:50:27.488514 5878 tablet.cc:2378] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:50:27.523195 5902 raft_consensus.cc:493] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:27.523433 5902 raft_consensus.cc:515] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.525082 5902 leader_election.cc:290] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6673ba6fdde845608628fadd4637f7dd (127.4.183.129:33093), 8eb4ae828a934d81b9b4bd485907d4d1 (127.4.183.131:37585)
I20260128 20:50:27.529935 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.525678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:33093 (local address 127.4.183.130:57953)
0128 20:50:27.526222 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:27.526246 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:27.526446 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:27.526954 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:27.526958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:27.527111 (+ 153us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:27.527686 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.527697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.528708 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.528711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:27.529450 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.529461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.529602 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.529625 (+ 23us) client_negotiation.cc:769] Sending connection context
0128 20:50:27.529681 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:50:27.529752 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":154,"threads_started":1}
I20260128 20:50:27.530400 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.525836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:57953 (local address 127.4.183.129:33093)
0128 20:50:27.526081 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.526089 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.526396 (+ 307us) server_negotiation.cc:408] Connection header received
0128 20:50:27.526629 (+ 233us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.526637 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.526707 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.526771 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:27.527832 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.528530 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.529580 (+ 1050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.530118 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.530161 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.530217 (+ 56us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.530279 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260128 20:50:27.531405 5559 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30008a07a9864c9fbe5b5571dea01e48" candidate_uuid: "6d349d30c30b4c82b17fdf98a576cd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6673ba6fdde845608628fadd4637f7dd" is_pre_election: true
I20260128 20:50:27.531731 5559 raft_consensus.cc:2468] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d349d30c30b4c82b17fdf98a576cd47 in term 0.
I20260128 20:50:27.531790 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.526486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:37585 (local address 127.4.183.130:38405)
0128 20:50:27.526958 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:27.527111 (+ 153us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:27.527290 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:27.528128 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:27.528132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:27.528169 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:27.528521 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.528528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.529902 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.529909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:27.531167 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:27.531185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.531364 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.531393 (+ 29us) client_negotiation.cc:769] Sending connection context
0128 20:50:27.531457 (+ 64us) client_negotiation.cc:240] Negotiation successful
0128 20:50:27.531571 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":147,"threads_started":1}
I20260128 20:50:27.532310 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.526641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:38405 (local address 127.4.183.131:37585)
0128 20:50:27.526809 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.526816 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.527284 (+ 468us) server_negotiation.cc:408] Connection header received
0128 20:50:27.527660 (+ 376us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.527666 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.527955 (+ 289us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.528101 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:27.528700 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.529686 (+ 986us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.531403 (+ 1717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.531982 (+ 579us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.532034 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.532104 (+ 70us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.532194 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:50:27.532353 5631 leader_election.cc:304] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6673ba6fdde845608628fadd4637f7dd, 6d349d30c30b4c82b17fdf98a576cd47; no voters:
I20260128 20:50:27.532756 5902 raft_consensus.cc:2804] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:27.532850 5902 raft_consensus.cc:493] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:27.532925 5902 raft_consensus.cc:3060] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:27.532936 5831 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30008a07a9864c9fbe5b5571dea01e48" candidate_uuid: "6d349d30c30b4c82b17fdf98a576cd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" is_pre_election: true
I20260128 20:50:27.533398 5831 raft_consensus.cc:2468] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d349d30c30b4c82b17fdf98a576cd47 in term 0.
I20260128 20:50:27.534515 5902 raft_consensus.cc:515] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.535063 5902 leader_election.cc:290] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [CANDIDATE]: Term 1 election: Requested vote from peers 6673ba6fdde845608628fadd4637f7dd (127.4.183.129:33093), 8eb4ae828a934d81b9b4bd485907d4d1 (127.4.183.131:37585)
I20260128 20:50:27.535650 5831 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30008a07a9864c9fbe5b5571dea01e48" candidate_uuid: "6d349d30c30b4c82b17fdf98a576cd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb4ae828a934d81b9b4bd485907d4d1"
I20260128 20:50:27.535622 5559 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30008a07a9864c9fbe5b5571dea01e48" candidate_uuid: "6d349d30c30b4c82b17fdf98a576cd47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6673ba6fdde845608628fadd4637f7dd"
I20260128 20:50:27.535777 5831 raft_consensus.cc:3060] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:27.535778 5559 raft_consensus.cc:3060] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:27.537102 5831 raft_consensus.cc:2468] T 30008a07a9864c9fbe5b5571dea01e48 P 8eb4ae828a934d81b9b4bd485907d4d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d349d30c30b4c82b17fdf98a576cd47 in term 1.
I20260128 20:50:27.537102 5559 raft_consensus.cc:2468] T 30008a07a9864c9fbe5b5571dea01e48 P 6673ba6fdde845608628fadd4637f7dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d349d30c30b4c82b17fdf98a576cd47 in term 1.
I20260128 20:50:27.537581 5631 leader_election.cc:304] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d349d30c30b4c82b17fdf98a576cd47, 8eb4ae828a934d81b9b4bd485907d4d1; no voters:
I20260128 20:50:27.537870 5902 raft_consensus.cc:2804] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:27.538208 5902 raft_consensus.cc:697] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [term 1 LEADER]: Becoming Leader. State: Replica: 6d349d30c30b4c82b17fdf98a576cd47, State: Running, Role: LEADER
I20260128 20:50:27.538671 5902 consensus_queue.cc:237] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } }
I20260128 20:50:27.543246 5410 catalog_manager.cc:5654] T 30008a07a9864c9fbe5b5571dea01e48 P 6d349d30c30b4c82b17fdf98a576cd47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d349d30c30b4c82b17fdf98a576cd47 (127.4.183.130). New cstate: current_term: 1 leader_uuid: "6d349d30c30b4c82b17fdf98a576cd47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6673ba6fdde845608628fadd4637f7dd" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d349d30c30b4c82b17fdf98a576cd47" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb4ae828a934d81b9b4bd485907d4d1" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37585 } health_report { overall_health: UNKNOWN } } }
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:27.603891 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.592266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.4.183.129:33093)
0128 20:50:27.592463 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.592470 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.592538 (+ 68us) server_negotiation.cc:408] Connection header received
0128 20:50:27.592666 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.592673 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.592750 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.592862 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:27.593867 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.594821 (+ 954us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.595635 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.595905 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.598523 (+ 2618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:27.598570 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:27.598593 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:27.598642 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:27.601626 (+ 2984us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.602245 (+ 619us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.602254 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.602259 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.602350 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.602659 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.602666 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.602671 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.602962 (+ 291us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:27.603244 (+ 282us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.603446 (+ 202us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.603628 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:27.637312 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.625122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.4.183.129:33093)
0128 20:50:27.625330 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.625335 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.625527 (+ 192us) server_negotiation.cc:408] Connection header received
0128 20:50:27.625689 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.625692 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.625743 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.625820 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:27.627214 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.627725 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.628805 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.628950 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.632702 (+ 3752us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:27.632737 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:27.632742 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:27.632796 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:27.634881 (+ 2085us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.635529 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.635538 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.635543 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.635673 (+ 130us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.636113 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.636122 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.636124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.636410 (+ 286us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:27.636583 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.636853 (+ 270us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.637028 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:27.669685 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:27.659189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.4.183.129:33093)
0128 20:50:27.659374 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:27.659378 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:27.659541 (+ 163us) server_negotiation.cc:408] Connection header received
0128 20:50:27.659711 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:27.659714 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:27.659767 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:27.659845 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:27.661385 (+ 1540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.661872 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:27.663005 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:27.663156 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:27.666318 (+ 3162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:27.666346 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:27.666348 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:27.666391 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:27.667966 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.668482 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.668488 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.668495 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.668547 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:27.668799 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:27.668803 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:27.668805 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:27.668969 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:27.669119 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:27.669365 (+ 246us) server_negotiation.cc:300] Negotiation successful
0128 20:50:27.669495 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260128 20:50:27.671662 5519 tablet_service.cc:3012] Rejecting scan request for tablet 30008a07a9864c9fbe5b5571dea01e48: Uninitialized: clean time has not yet been initialized
I20260128 20:50:27.674346 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5476
I20260128 20:50:27.681828 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5612
I20260128 20:50:27.688459 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5748
I20260128 20:50:27.694584 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5380
2026-01-28T20:50:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3898 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5919](info): set up 2 sockets
Jan 28 20:50:27 dist-test-slave-hss7 krb5kdc[5919](info): commencing operation
krb5kdc: starting...
W20260128 20:50:29.759409 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.039s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:29 dist-test-slave-hss7 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:29.925094 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39307
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:30.060572 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:30.060930 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:30.061036 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:30.065459 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:30.065662 5935 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260128 20:50:30.065702 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:30.065754 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:30.065806 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:30.065838 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:30.071571 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:39307
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:30.073005 5935 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:30.074236 5935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:30.081547 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:30.081570 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:30.082314 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:30.082854 5935 server_base.cc:1060] running on GCE node
I20260128 20:50:30.083565 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:30.084985 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:30.086282 5935 hybrid_clock.cc:648] HybridClock initialized: now 1769633430086133 us; error 144 us; skew 500 ppm
Jan 28 20:50:30 dist-test-slave-hss7 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633430, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:30.089743 5935 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:30.091305 5935 webserver.cc:492] Webserver started at http://127.4.183.190:41829/ using document root <none> and password file <none>
I20260128 20:50:30.091986 5935 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:30.092073 5935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:30.092334 5935 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:30.094533 5935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "4bb6c592c8d84650b44e7cb4187e6cf9"
format_stamp: "Formatted at 2026-01-28 20:50:30 on dist-test-slave-hss7"
server_key: "78d79c4b5317c0b5d6240c4ec54a6fa3"
server_key_iv: "8dac775f3cecffd013054fdbcbd6ecf8"
server_key_version: "encryptionkey@0"
I20260128 20:50:30.095113 5935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "4bb6c592c8d84650b44e7cb4187e6cf9"
format_stamp: "Formatted at 2026-01-28 20:50:30 on dist-test-slave-hss7"
server_key: "78d79c4b5317c0b5d6240c4ec54a6fa3"
server_key_iv: "8dac775f3cecffd013054fdbcbd6ecf8"
server_key_version: "encryptionkey@0"
I20260128 20:50:30.099525 5935 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:50:30.102722 5950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:30.104117 5935 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:50:30.104575 5935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "4bb6c592c8d84650b44e7cb4187e6cf9"
format_stamp: "Formatted at 2026-01-28 20:50:30 on dist-test-slave-hss7"
server_key: "78d79c4b5317c0b5d6240c4ec54a6fa3"
server_key_iv: "8dac775f3cecffd013054fdbcbd6ecf8"
server_key_version: "encryptionkey@0"
I20260128 20:50:30.104779 5935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:30.137920 5935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:30.141585 5935 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:30.141773 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:30.154078 5935 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:39307
I20260128 20:50:30.154484 6012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:39307 every 8 connection(s)
I20260128 20:50:30.155519 5935 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:30.158866 6013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:30.164711 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 5935
I20260128 20:50:30.164870 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:30.165257 4830 external_mini_cluster.cc:1468] Setting key 52fdb661793dea9ffc0e2664ef604589
I20260128 20:50:30.166436 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: Bootstrap starting.
I20260128 20:50:30.169126 6013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:30.170065 6013 log.cc:826] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:30.172374 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: No bootstrap required, opened a new log
Jan 28 20:50:30 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:30.176546 6013 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39307 } }
I20260128 20:50:30.176914 6013 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:30.177008 6013 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb6c592c8d84650b44e7cb4187e6cf9, State: Initialized, Role: FOLLOWER
I20260128 20:50:30.177609 6013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39307 } }
I20260128 20:50:30.177762 6013 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:30.177860 6013 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:30.177996 6013 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:30.181526 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:30.166629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.4.183.190:39307)
0128 20:50:30.167229 (+ 600us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:30.167240 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:30.167283 (+ 43us) server_negotiation.cc:408] Connection header received
0128 20:50:30.167942 (+ 659us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:30.167961 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:30.168333 (+ 372us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:30.168866 (+ 533us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:30.170243 (+ 1377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:30.171052 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:30.171807 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:30.172150 (+ 343us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:30.175278 (+ 3128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:30.175318 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:30.175338 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:30.175417 (+ 79us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:30.178425 (+ 3008us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:30.179003 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:30.179008 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:30.179014 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:30.179111 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:30.179385 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:30.179388 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:30.179392 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:30.179796 (+ 404us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:30.180069 (+ 273us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:30.180385 (+ 316us) server_negotiation.cc:300] Negotiation successful
0128 20:50:30.180673 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":233,"threads_started":1}
I20260128 20:50:30.244267 6013 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39307 } }
I20260128 20:50:30.244836 6013 leader_election.cc:304] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bb6c592c8d84650b44e7cb4187e6cf9; no voters:
I20260128 20:50:30.245297 6013 leader_election.cc:290] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:30.245482 6018 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:30.245759 6018 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb6c592c8d84650b44e7cb4187e6cf9, State: Running, Role: LEADER
I20260128 20:50:30.246222 6018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39307 } }
I20260128 20:50:30.246730 6013 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:30.248036 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb6c592c8d84650b44e7cb4187e6cf9. Latest consensus state: current_term: 1 leader_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39307 } } }
I20260128 20:50:30.248180 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:30.248553 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb6c592c8d84650b44e7cb4187e6cf9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39307 } } }
I20260128 20:50:30.248749 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:30.248835 6026 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:30.253876 6026 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:30.261837 6026 catalog_manager.cc:1357] Generated new cluster ID: f0f33c2d52a34d30b456309fda0029f7
I20260128 20:50:30.261972 6026 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:30.287545 6026 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:30.289870 6026 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:30.296912 6026 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: Generated new TSK 0
I20260128 20:50:30.297930 6026 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
W20260128 20:50:31.677783 4830 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.4.183.129: real 1.369s user 0.000s sys 0.004s
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:31.704456 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39307
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:31.825495 6041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:31.825836 6041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:31.825937 6041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:31.830288 6041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:31.830580 6041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:31.830852 6041 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:50:31.830976 6041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:31.837234 6041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:31.838661 6041 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:31.839700 6041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:31.848604 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:31.848636 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:31.848604 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:31.848905 6041 server_base.cc:1060] running on GCE node
I20260128 20:50:31.849753 6041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:31.850812 6041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:31.852147 6041 hybrid_clock.cc:648] HybridClock initialized: now 1769633431852098 us; error 73 us; skew 500 ppm
Jan 28 20:50:31 dist-test-slave-hss7 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633431, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:31.856840 6041 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:31.858453 6041 webserver.cc:492] Webserver started at http://127.4.183.129:37779/ using document root <none> and password file <none>
I20260128 20:50:31.859097 6041 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:31.859156 6041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:31.859354 6041 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:31.861361 6041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "ff1d6cb505304e6b82ea4aa96df1a275"
format_stamp: "Formatted at 2026-01-28 20:50:31 on dist-test-slave-hss7"
server_key: "81c6321a95165acfcd4b7f1b5dc2e4c9"
server_key_iv: "573d1db35d2fe2019d8243457a9ca553"
server_key_version: "encryptionkey@0"
I20260128 20:50:31.861886 6041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "ff1d6cb505304e6b82ea4aa96df1a275"
format_stamp: "Formatted at 2026-01-28 20:50:31 on dist-test-slave-hss7"
server_key: "81c6321a95165acfcd4b7f1b5dc2e4c9"
server_key_iv: "573d1db35d2fe2019d8243457a9ca553"
server_key_version: "encryptionkey@0"
I20260128 20:50:31.866336 6041 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:50:31.869349 6056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:31.870759 6041 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:31.870944 6041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "ff1d6cb505304e6b82ea4aa96df1a275"
format_stamp: "Formatted at 2026-01-28 20:50:31 on dist-test-slave-hss7"
server_key: "81c6321a95165acfcd4b7f1b5dc2e4c9"
server_key_iv: "573d1db35d2fe2019d8243457a9ca553"
server_key_version: "encryptionkey@0"
I20260128 20:50:31.871071 6041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:31.890585 6041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:31.899247 6041 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:31.899431 6041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:31.901054 6041 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:31.901192 6041 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260128 20:50:31.901283 6041 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:31.901309 6041 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:50:31 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633431, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:31.915071 6041 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:33947
I20260128 20:50:31.915092 6172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:33947 every 8 connection(s)
I20260128 20:50:31.916236 6041 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:50:31.916647 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:31.901867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:38253 (local address 127.4.183.190:39307)
0128 20:50:31.902154 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:31.902158 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:31.903422 (+ 1264us) server_negotiation.cc:408] Connection header received
0128 20:50:31.904617 (+ 1195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:31.904622 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:31.904698 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:31.904801 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:31.906601 (+ 1800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.907168 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:31.908217 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.908512 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:31.911678 (+ 3166us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:31.911709 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:31.911712 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:31.911757 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:31.914106 (+ 2349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:31.914905 (+ 799us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:31.914910 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:31.914912 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:31.914982 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:31.915388 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:31.915395 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:31.915399 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:31.915597 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:31.915725 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:31.916289 (+ 564us) server_negotiation.cc:300] Negotiation successful
0128 20:50:31.916451 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":112,"threads_started":1}
I20260128 20:50:31.917493 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:31.902284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.4.183.129:38253)
0128 20:50:31.903171 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:31.903260 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:31.904330 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:31.905089 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:31.905099 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:31.905658 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:31.906343 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:31.906378 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.907331 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:31.907335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:31.908026 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:31.908037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.908392 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:31.909237 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:31.909273 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:31.911400 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:31.914343 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:31.914349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:31.914361 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:31.914669 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:31.915110 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:31.915113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:31.915115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:31.915280 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:31.915725 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:31.915734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:31.916013 (+ 279us) client_negotiation.cc:769] Sending connection context
0128 20:50:31.916246 (+ 233us) client_negotiation.cc:240] Negotiation successful
0128 20:50:31.916570 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":185,"threads_started":1}
I20260128 20:50:31.918866 6173 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39307
I20260128 20:50:31.919075 6173 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:31.919710 6173 heartbeater.cc:507] Master 127.4.183.190:39307 requested a full tablet report, sending...
I20260128 20:50:31.921636 5965 ts_manager.cc:194] Registered new tserver with Master: ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947)
I20260128 20:50:31.923383 5965 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:38253
I20260128 20:50:31.924240 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6041
I20260128 20:50:31.924355 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:31.924621 4830 external_mini_cluster.cc:1468] Setting key abec1830bf3c70e5e761553177e8cee3
I20260128 20:50:31.933867 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:31.921739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:38429 (local address 127.4.183.190:39307)
0128 20:50:31.921941 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:31.921946 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:31.922263 (+ 317us) server_negotiation.cc:408] Connection header received
0128 20:50:31.922378 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:31.922381 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:31.922435 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:31.922526 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:31.923393 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.924131 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:31.927051 (+ 2920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.927340 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:31.928504 (+ 1164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:31.928536 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:31.928539 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:31.928566 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:31.931017 (+ 2451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:31.931945 (+ 928us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:31.931953 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:31.931956 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:31.932080 (+ 124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:31.932674 (+ 594us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:31.932678 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:31.932680 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:31.932898 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:31.933022 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:31.933492 (+ 470us) server_negotiation.cc:300] Negotiation successful
0128 20:50:31.933684 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260128 20:50:31.933902 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:31.921646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.4.183.129:38429)
0128 20:50:31.922106 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:31.922126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:31.922273 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:31.922576 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:31.922579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:31.922823 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:31.923142 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:31.923151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.924331 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:31.924337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:31.926889 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:31.926904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:31.927173 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:31.927924 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:31.927939 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:31.928339 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:31.931319 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:31.931328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:31.931334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:31.931669 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:31.932310 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:31.932318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:31.932323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:31.932454 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:31.933033 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:31.933038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:31.933214 (+ 176us) client_negotiation.cc:769] Sending connection context
0128 20:50:31.933367 (+ 153us) client_negotiation.cc:240] Negotiation successful
0128 20:50:31.933585 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":197,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:31.989437 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39307
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:32.117372 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:32.117623 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:32.117688 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:32.121239 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:32.121325 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:32.121382 6184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:50:32.121426 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:32.126606 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:32.127756 6184 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:32.128674 6184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:32.136314 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:32.136529 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:32.136315 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:32.136725 6184 server_base.cc:1060] running on GCE node
I20260128 20:50:32.137329 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:32.137972 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:32.139271 6184 hybrid_clock.cc:648] HybridClock initialized: now 1769633432139164 us; error 48 us; skew 500 ppm
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:32.142637 6184 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:32.143924 6184 webserver.cc:492] Webserver started at http://127.4.183.130:42773/ using document root <none> and password file <none>
I20260128 20:50:32.144568 6184 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:32.144652 6184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:32.144884 6184 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:32.146885 6184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "04750e3db41b4677b755e1cf3362917f"
format_stamp: "Formatted at 2026-01-28 20:50:32 on dist-test-slave-hss7"
server_key: "c3b6aaf3f836d7daf5efb61cd4685c76"
server_key_iv: "71c6ac5f4e7c04bc7537d7bb7e511674"
server_key_version: "encryptionkey@0"
I20260128 20:50:32.147501 6184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "04750e3db41b4677b755e1cf3362917f"
format_stamp: "Formatted at 2026-01-28 20:50:32 on dist-test-slave-hss7"
server_key: "c3b6aaf3f836d7daf5efb61cd4685c76"
server_key_iv: "71c6ac5f4e7c04bc7537d7bb7e511674"
server_key_version: "encryptionkey@0"
I20260128 20:50:32.151603 6184 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260128 20:50:32.154830 6199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:32.156473 6184 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:50:32.156663 6184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "04750e3db41b4677b755e1cf3362917f"
format_stamp: "Formatted at 2026-01-28 20:50:32 on dist-test-slave-hss7"
server_key: "c3b6aaf3f836d7daf5efb61cd4685c76"
server_key_iv: "71c6ac5f4e7c04bc7537d7bb7e511674"
server_key_version: "encryptionkey@0"
I20260128 20:50:32.156795 6184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:32.181102 6184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:32.184489 6184 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:32.184728 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:32.186321 6184 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:32.186420 6184 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:32.186493 6184 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:32.186543 6184 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:32.199146 6184 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:38203
I20260128 20:50:32.199172 6314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:38203 every 8 connection(s)
I20260128 20:50:32.200433 6184 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:32.200744 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.187230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:35517 (local address 127.4.183.190:39307)
0128 20:50:32.187408 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.187416 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.188297 (+ 881us) server_negotiation.cc:408] Connection header received
0128 20:50:32.189539 (+ 1242us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.189543 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.189646 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.189751 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:32.191535 (+ 1784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.192104 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.192821 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.193002 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.195602 (+ 2600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:32.195626 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:32.195629 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:32.195661 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:32.197629 (+ 1968us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.198235 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.198239 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.198241 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.198300 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.199015 (+ 715us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.199019 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.199021 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.199536 (+ 515us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:32.199649 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.200380 (+ 731us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.200588 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260128 20:50:32.201525 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.187521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.4.183.130:35517)
0128 20:50:32.188102 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.188163 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.189230 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.190069 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.190088 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.190690 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:32.191327 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.191349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.192251 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.192256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.192686 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.192693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.192963 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.193701 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:32.193759 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:32.195392 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:32.197820 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.197830 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.197848 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.198116 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.198533 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.198539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.198541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.198735 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.199681 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:32.199689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:32.200023 (+ 334us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.200299 (+ 276us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.200616 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":160,"threads_started":1}
I20260128 20:50:32.202960 6315 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39307
I20260128 20:50:32.203218 6315 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:32.203734 6315 heartbeater.cc:507] Master 127.4.183.190:39307 requested a full tablet report, sending...
I20260128 20:50:32.204908 5967 ts_manager.cc:194] Registered new tserver with Master: 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203)
I20260128 20:50:32.205710 5967 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:35517
I20260128 20:50:32.207549 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6184
I20260128 20:50:32.207648 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:32.207842 4830 external_mini_cluster.cc:1468] Setting key e99c80d9d21cfdf0dfc59c36fe42765c
I20260128 20:50:32.213873 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.205405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:47701 (local address 127.4.183.190:39307)
0128 20:50:32.205549 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.205554 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.205758 (+ 204us) server_negotiation.cc:408] Connection header received
0128 20:50:32.205899 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.205903 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.205947 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.206154 (+ 207us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:32.207103 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.207845 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.208771 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.209029 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.210006 (+ 977us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:32.210026 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:32.210029 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:32.210058 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:32.211887 (+ 1829us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.212356 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.212362 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.212364 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.212420 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.212667 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.212671 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.212673 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.212893 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:32.213016 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.213570 (+ 554us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.213702 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 20:50:32.213896 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.205384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.4.183.130:47701)
0128 20:50:32.205666 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.205680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.205780 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.206456 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.206459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.206710 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:32.206944 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.206950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.208006 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.208010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.208519 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.208529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.208729 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.209422 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:32.209455 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:32.209828 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:32.212025 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.212029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.212031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.212242 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.212520 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.212528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.212530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.212576 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.212994 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:32.212997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:32.213274 (+ 277us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.213511 (+ 237us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.213693 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:32.269490 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39307
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:32.409288 6326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:32.409680 6326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:32.409760 6326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:32.414273 6326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:32.414391 6326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:32.414461 6326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:50:32.414511 6326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:32.421384 6326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:32.422744 6326 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:32.423717 6326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:32.431941 6334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:32.432039 6331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:32.431969 6332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:32.432248 6326 server_base.cc:1060] running on GCE node
I20260128 20:50:32.432829 6326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:32.433607 6326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:32.434893 6326 hybrid_clock.cc:648] HybridClock initialized: now 1769633432434834 us; error 65 us; skew 500 ppm
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:32.439730 6326 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:32.441118 6326 webserver.cc:492] Webserver started at http://127.4.183.131:35437/ using document root <none> and password file <none>
I20260128 20:50:32.441847 6326 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:32.441934 6326 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:32.442179 6326 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:32.444109 6326 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "d7d86a988796472c988683396093ddbb"
format_stamp: "Formatted at 2026-01-28 20:50:32 on dist-test-slave-hss7"
server_key: "d7d96b8ff1ffc41f347b305e63624249"
server_key_iv: "91c93f39aa599b99a4c59fd048f17c05"
server_key_version: "encryptionkey@0"
I20260128 20:50:32.444720 6326 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d86a988796472c988683396093ddbb"
format_stamp: "Formatted at 2026-01-28 20:50:32 on dist-test-slave-hss7"
server_key: "d7d96b8ff1ffc41f347b305e63624249"
server_key_iv: "91c93f39aa599b99a4c59fd048f17c05"
server_key_version: "encryptionkey@0"
I20260128 20:50:32.448999 6326 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260128 20:50:32.453011 6341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:32.454576 6326 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:32.454810 6326 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "d7d86a988796472c988683396093ddbb"
format_stamp: "Formatted at 2026-01-28 20:50:32 on dist-test-slave-hss7"
server_key: "d7d96b8ff1ffc41f347b305e63624249"
server_key_iv: "91c93f39aa599b99a4c59fd048f17c05"
server_key_version: "encryptionkey@0"
I20260128 20:50:32.454955 6326 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:32.480018 6326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:32.483614 6326 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:32.483870 6326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:32.485669 6326 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:32.485756 6326 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:32.485836 6326 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:32.485893 6326 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:32.501704 6326 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:43283
I20260128 20:50:32.502840 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.486434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:43587 (local address 127.4.183.190:39307)
0128 20:50:32.486607 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.486612 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.487384 (+ 772us) server_negotiation.cc:408] Connection header received
0128 20:50:32.488684 (+ 1300us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.488688 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.488749 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.488797 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:32.490792 (+ 1995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.491350 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.492551 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.492869 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.496931 (+ 4062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:32.496978 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:32.496984 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:32.497029 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:32.499841 (+ 2812us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.500700 (+ 859us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.500705 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.500730 (+ 25us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.500828 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.501349 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.501354 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.501356 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.501602 (+ 246us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:32.501713 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.502494 (+ 781us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.502657 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:50:32.503036 6326 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:32.503494 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.486715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.4.183.131:43587)
0128 20:50:32.487204 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.487252 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.488287 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.489234 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.489243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.489893 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:32.490593 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.490608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.491493 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.491499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.492262 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.492285 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.492768 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.493903 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:32.493983 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:32.496627 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:32.500065 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.500076 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.500097 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.500547 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.501009 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.501015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.501017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.501215 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.501778 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:32.501785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:32.502133 (+ 348us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.502406 (+ 273us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.502682 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":150,"threads_started":1}
I20260128 20:50:32.503835 6456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:43283 every 8 connection(s)
I20260128 20:50:32.505290 6457 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39307
I20260128 20:50:32.505496 6457 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:32.506088 6457 heartbeater.cc:507] Master 127.4.183.190:39307 requested a full tablet report, sending...
I20260128 20:50:32.507318 5967 ts_manager.cc:194] Registered new tserver with Master: d7d86a988796472c988683396093ddbb (127.4.183.131:43283)
I20260128 20:50:32.507766 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6326
I20260128 20:50:32.507889 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:32.508104 5967 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:43587
I20260128 20:50:32.508134 4830 external_mini_cluster.cc:1468] Setting key fdf341a5dbd5ee351e511a7449486863
I20260128 20:50:32.510789 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 20:50:32.516078 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.507719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.4.183.131:55757)
0128 20:50:32.508579 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.508593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.508721 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.509012 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.509016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.509419 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:32.509711 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.509721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.510514 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.510518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.511007 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.511016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.511155 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.511822 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:32.511841 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:32.512234 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:32.514049 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.514053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.514055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.514296 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.514622 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.514628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.514630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.514695 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.515292 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:32.515296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:32.515463 (+ 167us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.515636 (+ 173us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.515872 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":109,"threads_started":1}
I20260128 20:50:32.516096 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.507771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:55757 (local address 127.4.183.190:39307)
0128 20:50:32.507894 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.507898 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.508671 (+ 773us) server_negotiation.cc:408] Connection header received
0128 20:50:32.508820 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.508823 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.508895 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.508974 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:32.509846 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.510388 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.511127 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.511331 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.512365 (+ 1034us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:32.512397 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:32.512399 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:32.512423 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:32.513905 (+ 1482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.514434 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.514438 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.514440 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.514497 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.514881 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.514892 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.514897 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.515167 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:32.515279 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.515708 (+ 429us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.515856 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:32.542563 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.531350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.4.183.190:39307)
0128 20:50:32.531525 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.531529 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.531544 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:50:32.531591 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.531594 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.531640 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.531728 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:32.533106 (+ 1378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.534015 (+ 909us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.535007 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.535227 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.538435 (+ 3208us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:32.538463 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:32.538466 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:32.538511 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:32.540263 (+ 1752us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.540996 (+ 733us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.541001 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.541003 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.541085 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.541598 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.541601 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.541602 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.541796 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:32.541926 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.542268 (+ 342us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.542417 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260128 20:50:32.546466 5967 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:50:32.549413 5967 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:32.567734 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.561337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:43283 (local address 127.0.0.1:55086)
0128 20:50:32.562616 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.562662 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.562887 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.563727 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.563733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.563760 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.564077 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.564089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.566216 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.566222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.567068 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.567077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.567229 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.567317 (+ 88us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.567437 (+ 120us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.567562 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":155,"threads_started":1}
I20260128 20:50:32.568303 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.561602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.4.183.131:43283)
0128 20:50:32.562311 (+ 709us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.562319 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.562754 (+ 435us) server_negotiation.cc:408] Connection header received
0128 20:50:32.563081 (+ 327us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.563089 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.563316 (+ 227us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.563466 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.564220 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.566098 (+ 1878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.567234 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.567773 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.567916 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.567988 (+ 72us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.568091 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":159,"threads_started":1}
I20260128 20:50:32.568540 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.561591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:38203 (local address 127.0.0.1:50168)
0128 20:50:32.563318 (+ 1727us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.563390 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.563501 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.564365 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.564372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.564396 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.564683 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.564691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.566555 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.566560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.567622 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.567637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.567743 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.567765 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.568098 (+ 333us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.568184 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1628,"thread_start_us":125,"threads_started":1}
I20260128 20:50:32.571102 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.562296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.4.183.130:38203)
0128 20:50:32.563847 (+ 1551us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.563855 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.563882 (+ 27us) server_negotiation.cc:408] Connection header received
0128 20:50:32.563972 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.563980 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.564170 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.564856 (+ 686us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.565045 (+ 189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.566387 (+ 1342us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.569076 (+ 2689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.570198 (+ 1122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.570579 (+ 381us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.570686 (+ 107us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.570844 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":153,"threads_started":1}
I20260128 20:50:32.571653 6391 tablet_service.cc:1505] Processing CreateTablet for tablet a59301d9f7914ec29350ee6007f1302d (DEFAULT_TABLE table=test-table [id=1dafe6b050e1438f8643d4c4274acc74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:32.572764 6391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59301d9f7914ec29350ee6007f1302d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:32.574654 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.563763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:33947 (local address 127.0.0.1:51840)
0128 20:50:32.564787 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.564802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.564887 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.565615 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.565622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.565646 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.565915 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.565924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.572623 (+ 6699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.572631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.574024 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.574047 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.574211 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.574246 (+ 35us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.574326 (+ 80us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.574432 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":159,"threads_started":1}
I20260128 20:50:32.574424 6249 tablet_service.cc:1505] Processing CreateTablet for tablet a59301d9f7914ec29350ee6007f1302d (DEFAULT_TABLE table=test-table [id=1dafe6b050e1438f8643d4c4274acc74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:32.575763 6249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59301d9f7914ec29350ee6007f1302d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:32.576663 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.564458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.4.183.129:33947)
0128 20:50:32.565152 (+ 694us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.565160 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.565183 (+ 23us) server_negotiation.cc:408] Connection header received
0128 20:50:32.565266 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.565272 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.565461 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.566063 (+ 602us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.568173 (+ 2110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.572377 (+ 4204us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.575480 (+ 3103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.576170 (+ 690us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.576315 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.576410 (+ 95us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.576529 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":78,"threads_started":1}
I20260128 20:50:32.579617 6107 tablet_service.cc:1505] Processing CreateTablet for tablet a59301d9f7914ec29350ee6007f1302d (DEFAULT_TABLE table=test-table [id=1dafe6b050e1438f8643d4c4274acc74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:32.580909 6107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59301d9f7914ec29350ee6007f1302d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:32.583067 6481 tablet_bootstrap.cc:492] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Bootstrap starting.
I20260128 20:50:32.585099 6482 tablet_bootstrap.cc:492] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Bootstrap starting.
I20260128 20:50:32.586714 6481 tablet_bootstrap.cc:654] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:32.588505 6481 log.cc:826] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:32.590579 6482 tablet_bootstrap.cc:654] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:32.592008 6484 tablet_bootstrap.cc:492] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Bootstrap starting.
I20260128 20:50:32.592243 6482 log.cc:826] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:32.592446 6481 tablet_bootstrap.cc:492] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: No bootstrap required, opened a new log
I20260128 20:50:32.592695 6481 ts_tablet_manager.cc:1403] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260128 20:50:32.594987 6484 tablet_bootstrap.cc:654] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:32.595139 6482 tablet_bootstrap.cc:492] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: No bootstrap required, opened a new log
I20260128 20:50:32.595403 6482 ts_tablet_manager.cc:1403] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260128 20:50:32.595994 6484 log.cc:826] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:32.597388 6481 raft_consensus.cc:359] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.597659 6481 raft_consensus.cc:385] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:32.597723 6481 raft_consensus.cc:740] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d86a988796472c988683396093ddbb, State: Initialized, Role: FOLLOWER
I20260128 20:50:32.598282 6481 consensus_queue.cc:260] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.598505 6484 tablet_bootstrap.cc:492] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: No bootstrap required, opened a new log
I20260128 20:50:32.598757 6484 ts_tablet_manager.cc:1403] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260128 20:50:32.599454 6481 ts_tablet_manager.cc:1434] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260128 20:50:32.599570 6482 raft_consensus.cc:359] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.599941 6482 raft_consensus.cc:385] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:32.600000 6482 raft_consensus.cc:740] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04750e3db41b4677b755e1cf3362917f, State: Initialized, Role: FOLLOWER
I20260128 20:50:32.600014 6457 heartbeater.cc:499] Master 127.4.183.190:39307 was elected leader, sending a full tablet report...
I20260128 20:50:32.600629 6482 consensus_queue.cc:260] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.601796 6315 heartbeater.cc:499] Master 127.4.183.190:39307 was elected leader, sending a full tablet report...
I20260128 20:50:32.601999 6482 ts_tablet_manager.cc:1434] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260128 20:50:32.602344 6484 raft_consensus.cc:359] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.602636 6484 raft_consensus.cc:385] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:32.602713 6484 raft_consensus.cc:740] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1d6cb505304e6b82ea4aa96df1a275, State: Initialized, Role: FOLLOWER
I20260128 20:50:32.603291 6484 consensus_queue.cc:260] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.604601 6484 ts_tablet_manager.cc:1434] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260128 20:50:32.605044 6173 heartbeater.cc:499] Master 127.4.183.190:39307 was elected leader, sending a full tablet report...
I20260128 20:50:32.622675 6487 raft_consensus.cc:493] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:32.622895 6487 raft_consensus.cc:515] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.624349 6487 leader_election.cc:290] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203), ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947)
I20260128 20:50:32.628427 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.624817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:38203 (local address 127.4.183.131:56239)
0128 20:50:32.625087 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.625108 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.625288 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.625683 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.625686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.625707 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.626137 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.626163 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.627193 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.627197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.627843 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.627852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.628051 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.628080 (+ 29us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.628151 (+ 71us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.628235 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260128 20:50:32.628893 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.625005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:56239 (local address 127.4.183.130:38203)
0128 20:50:32.625233 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.625238 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.625293 (+ 55us) server_negotiation.cc:408] Connection header received
0128 20:50:32.625462 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.625469 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.625544 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.625643 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.626456 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.627031 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.628021 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.628555 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.628601 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.628666 (+ 65us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.628743 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:50:32.629624 6269 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a59301d9f7914ec29350ee6007f1302d" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04750e3db41b4677b755e1cf3362917f" is_pre_election: true
I20260128 20:50:32.629707 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.625295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:33947 (local address 127.4.183.131:39583)
0128 20:50:32.625797 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.625811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.625934 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.626206 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.626210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.626226 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.626532 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.626539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.627755 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.627760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.628642 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.628655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.629395 (+ 740us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.629418 (+ 23us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.629489 (+ 71us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.629558 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":192,"threads_started":1}
I20260128 20:50:32.629693 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.625465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:39583 (local address 127.4.183.129:33947)
0128 20:50:32.625649 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.625654 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.625900 (+ 246us) server_negotiation.cc:408] Connection header received
0128 20:50:32.626043 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.626048 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.626101 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.626194 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.626658 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.627477 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.628834 (+ 1357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.629414 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.629458 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.629519 (+ 61us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.629585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260128 20:50:32.629984 6269 raft_consensus.cc:2468] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 0.
I20260128 20:50:32.630586 6342 leader_election.cc:304] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04750e3db41b4677b755e1cf3362917f, d7d86a988796472c988683396093ddbb; no voters:
I20260128 20:50:32.630847 6127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a59301d9f7914ec29350ee6007f1302d" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" is_pre_election: true
I20260128 20:50:32.630905 6487 raft_consensus.cc:2804] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:32.630997 6487 raft_consensus.cc:493] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:32.631067 6487 raft_consensus.cc:3060] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:32.631188 6127 raft_consensus.cc:2468] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 0.
I20260128 20:50:32.632293 6487 raft_consensus.cc:515] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.632725 6487 leader_election.cc:290] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 election: Requested vote from peers 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203), ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947)
I20260128 20:50:32.633293 6269 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a59301d9f7914ec29350ee6007f1302d" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04750e3db41b4677b755e1cf3362917f"
I20260128 20:50:32.633481 6269 raft_consensus.cc:3060] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:32.633436 6127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a59301d9f7914ec29350ee6007f1302d" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1d6cb505304e6b82ea4aa96df1a275"
I20260128 20:50:32.633638 6127 raft_consensus.cc:3060] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:32.634769 6269 raft_consensus.cc:2468] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 1.
I20260128 20:50:32.634778 6127 raft_consensus.cc:2468] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 1.
I20260128 20:50:32.635254 6342 leader_election.cc:304] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d86a988796472c988683396093ddbb, ff1d6cb505304e6b82ea4aa96df1a275; no voters:
I20260128 20:50:32.635629 6487 raft_consensus.cc:2804] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:32.636229 6487 raft_consensus.cc:697] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 1 LEADER]: Becoming Leader. State: Replica: d7d86a988796472c988683396093ddbb, State: Running, Role: LEADER
I20260128 20:50:32.636672 6487 consensus_queue.cc:237] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } }
I20260128 20:50:32.640919 5967 catalog_manager.cc:5654] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d86a988796472c988683396093ddbb (127.4.183.131). New cstate: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } health_report { overall_health: UNKNOWN } } }
I20260128 20:50:32.660286 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.656014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.4.183.131:43283)
0128 20:50:32.656379 (+ 365us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.656403 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.656423 (+ 20us) server_negotiation.cc:408] Connection header received
0128 20:50:32.656700 (+ 277us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.656704 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.656775 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.656881 (+ 106us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:32.657445 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.658086 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.658809 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.659001 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.659083 (+ 82us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:32.659658 (+ 575us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:32.659815 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.659996 (+ 181us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.660066 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
W20260128 20:50:32.668216 6174 tablet.cc:2378] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:50:32.668697 6127 raft_consensus.cc:1275] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Refusing update from remote peer d7d86a988796472c988683396093ddbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:32.668697 6269 raft_consensus.cc:1275] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Refusing update from remote peer d7d86a988796472c988683396093ddbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:32.669835 6487 consensus_queue.cc:1048] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:32.670156 6491 consensus_queue.cc:1048] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:32.681598 6495 mvcc.cc:204] Tried to move back new op lower bound from 7248418540197801984 to 7248418540083687424. Current Snapshot: MvccSnapshot[applied={T|T < 7248418540197801984}]
I20260128 20:50:32.696995 5966 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:32.697290 5966 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:32.701869 5966 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: Sending DeleteTablet for 3 replicas of tablet a59301d9f7914ec29350ee6007f1302d
I20260128 20:50:32.702924 6391 tablet_service.cc:1552] Processing DeleteTablet for tablet a59301d9f7914ec29350ee6007f1302d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:32 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:55086
I20260128 20:50:32.703148 6249 tablet_service.cc:1552] Processing DeleteTablet for tablet a59301d9f7914ec29350ee6007f1302d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:32 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:50168
I20260128 20:50:32.703542 6504 tablet_replica.cc:333] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: stopping tablet replica
I20260128 20:50:32.703908 6504 raft_consensus.cc:2243] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:50:32.704017 6107 tablet_service.cc:1552] Processing DeleteTablet for tablet a59301d9f7914ec29350ee6007f1302d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:32 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:51840
I20260128 20:50:32.704591 6504 raft_consensus.cc:2272] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:32.704715 6505 tablet_replica.cc:333] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: stopping tablet replica
I20260128 20:50:32.704697 5966 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 20:50:32.705026 6505 raft_consensus.cc:2243] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260128 20:50:32.705312 5966 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:32.705468 6505 raft_consensus.cc:2272] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:32.705487 6506 tablet_replica.cc:333] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: stopping tablet replica
I20260128 20:50:32.705981 6506 raft_consensus.cc:2243] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:32.706389 6506 raft_consensus.cc:2272] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:32.707887 6506 ts_tablet_manager.cc:1916] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:32.707887 6505 ts_tablet_manager.cc:1916] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:32.710898 6506 ts_tablet_manager.cc:1929] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:50:32.710898 6505 ts_tablet_manager.cc:1929] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:50:32.711023 6505 log.cc:1199] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/a59301d9f7914ec29350ee6007f1302d
I20260128 20:50:32.711200 6506 log.cc:1199] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/a59301d9f7914ec29350ee6007f1302d
I20260128 20:50:32.711366 6505 ts_tablet_manager.cc:1950] T a59301d9f7914ec29350ee6007f1302d P 04750e3db41b4677b755e1cf3362917f: Deleting consensus metadata
I20260128 20:50:32.711656 6506 ts_tablet_manager.cc:1950] T a59301d9f7914ec29350ee6007f1302d P ff1d6cb505304e6b82ea4aa96df1a275: Deleting consensus metadata
I20260128 20:50:32.712937 6391 tablet_service.cc:1505] Processing CreateTablet for tablet 11c3a82ba87f42a7add2b357f095955c (DEFAULT_TABLE table=test-table [id=91f97408b6084f0c9a35de69fb3be2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:32.713341 6391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c3a82ba87f42a7add2b357f095955c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:32.714069 5951 catalog_manager.cc:4985] TS 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203): tablet a59301d9f7914ec29350ee6007f1302d (table test-table [id=1dafe6b050e1438f8643d4c4274acc74]) successfully deleted
I20260128 20:50:32.714443 5951 catalog_manager.cc:4985] TS ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947): tablet a59301d9f7914ec29350ee6007f1302d (table test-table [id=1dafe6b050e1438f8643d4c4274acc74]) successfully deleted
I20260128 20:50:32.715248 6249 tablet_service.cc:1505] Processing CreateTablet for tablet 11c3a82ba87f42a7add2b357f095955c (DEFAULT_TABLE table=test-table [id=91f97408b6084f0c9a35de69fb3be2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:32.715641 6249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c3a82ba87f42a7add2b357f095955c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:32.716388 6107 tablet_service.cc:1505] Processing CreateTablet for tablet 11c3a82ba87f42a7add2b357f095955c (DEFAULT_TABLE table=test-table [id=91f97408b6084f0c9a35de69fb3be2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:32.716689 6107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c3a82ba87f42a7add2b357f095955c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:32.717111 6504 ts_tablet_manager.cc:1916] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:32.719944 6482 tablet_bootstrap.cc:492] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Bootstrap starting.
I20260128 20:50:32.720079 6504 ts_tablet_manager.cc:1929] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:50:32.720237 6504 log.cc:1199] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/a59301d9f7914ec29350ee6007f1302d
I20260128 20:50:32.720562 6504 ts_tablet_manager.cc:1950] T a59301d9f7914ec29350ee6007f1302d P d7d86a988796472c988683396093ddbb: Deleting consensus metadata
I20260128 20:50:32.721258 6482 tablet_bootstrap.cc:654] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:32.721422 6484 tablet_bootstrap.cc:492] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Bootstrap starting.
I20260128 20:50:32.722105 5953 catalog_manager.cc:4985] TS d7d86a988796472c988683396093ddbb (127.4.183.131:43283): tablet a59301d9f7914ec29350ee6007f1302d (table test-table [id=1dafe6b050e1438f8643d4c4274acc74]) successfully deleted
I20260128 20:50:32.722720 6484 tablet_bootstrap.cc:654] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:32.722764 6481 tablet_bootstrap.cc:492] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Bootstrap starting.
I20260128 20:50:32.723907 6481 tablet_bootstrap.cc:654] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:32.724133 6484 tablet_bootstrap.cc:492] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: No bootstrap required, opened a new log
I20260128 20:50:32.724238 6484 ts_tablet_manager.cc:1403] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260128 20:50:32.724818 6484 raft_consensus.cc:359] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.724954 6484 raft_consensus.cc:385] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:32.724995 6484 raft_consensus.cc:740] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1d6cb505304e6b82ea4aa96df1a275, State: Initialized, Role: FOLLOWER
I20260128 20:50:32.725253 6484 consensus_queue.cc:260] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.725908 6482 tablet_bootstrap.cc:492] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: No bootstrap required, opened a new log
I20260128 20:50:32.726011 6482 ts_tablet_manager.cc:1403] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260128 20:50:32.726120 6484 ts_tablet_manager.cc:1434] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260128 20:50:32.726539 6482 raft_consensus.cc:359] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.726679 6482 raft_consensus.cc:385] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:32.726711 6482 raft_consensus.cc:740] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04750e3db41b4677b755e1cf3362917f, State: Initialized, Role: FOLLOWER
I20260128 20:50:32.726846 6482 consensus_queue.cc:260] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.727164 6482 ts_tablet_manager.cc:1434] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 20:50:32.728797 6481 tablet_bootstrap.cc:492] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: No bootstrap required, opened a new log
I20260128 20:50:32.728921 6481 ts_tablet_manager.cc:1403] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260128 20:50:32.729517 6481 raft_consensus.cc:359] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.729657 6481 raft_consensus.cc:385] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:32.729691 6481 raft_consensus.cc:740] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d86a988796472c988683396093ddbb, State: Initialized, Role: FOLLOWER
I20260128 20:50:32.729854 6481 consensus_queue.cc:260] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.730252 6481 ts_tablet_manager.cc:1434] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260128 20:50:32.754070 6458 tablet.cc:2378] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:50:32.902839 6489 raft_consensus.cc:493] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:32.903029 6489 raft_consensus.cc:515] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.904564 6489 leader_election.cc:290] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203), d7d86a988796472c988683396093ddbb (127.4.183.131:43283)
I20260128 20:50:32.909691 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.904896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:43283 (local address 127.4.183.129:60375)
0128 20:50:32.905375 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.905392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.905544 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.905913 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.905916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.905958 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.906290 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.906296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.907698 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.907703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.909027 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.909046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.909297 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.909329 (+ 32us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.909383 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.909464 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"spinlock_wait_cycles":9856,"thread_start_us":97,"threads_started":1}
I20260128 20:50:32.910113 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.904894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:38203 (local address 127.4.183.129:56971)
0128 20:50:32.905449 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.905463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.905554 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.905872 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.905878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.905907 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:32.906268 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.906276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.907698 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.907703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.909471 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.909480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.909610 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.909626 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.909676 (+ 50us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.909738 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":196,"spinlock_wait_cycles":1792,"thread_start_us":118,"threads_started":1}
I20260128 20:50:32.910231 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.905104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:60375 (local address 127.4.183.131:43283)
0128 20:50:32.905300 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.905305 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.905536 (+ 231us) server_negotiation.cc:408] Connection header received
0128 20:50:32.905658 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.905661 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.905719 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.905814 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.906544 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.907444 (+ 900us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.909290 (+ 1846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.909878 (+ 588us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.909926 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.909983 (+ 57us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.910055 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260128 20:50:32.910480 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.905025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:56971 (local address 127.4.183.130:38203)
0128 20:50:32.905234 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.905241 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.905557 (+ 316us) server_negotiation.cc:408] Connection header received
0128 20:50:32.905656 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.905660 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.905719 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.905814 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:32.906547 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.907444 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.909667 (+ 2223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.910166 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.910204 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.910263 (+ 59us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.910336 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260128 20:50:32.910957 6411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11c3a82ba87f42a7add2b357f095955c" candidate_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d86a988796472c988683396093ddbb" is_pre_election: true
I20260128 20:50:32.911015 6269 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11c3a82ba87f42a7add2b357f095955c" candidate_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04750e3db41b4677b755e1cf3362917f" is_pre_election: true
I20260128 20:50:32.911163 6269 raft_consensus.cc:2468] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1d6cb505304e6b82ea4aa96df1a275 in term 0.
I20260128 20:50:32.911213 6411 raft_consensus.cc:2468] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1d6cb505304e6b82ea4aa96df1a275 in term 0.
I20260128 20:50:32.911666 6059 leader_election.cc:304] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d86a988796472c988683396093ddbb, ff1d6cb505304e6b82ea4aa96df1a275; no voters:
I20260128 20:50:32.911975 6489 raft_consensus.cc:2804] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:32.912038 6489 raft_consensus.cc:493] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:32.912077 6489 raft_consensus.cc:3060] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:32.913057 6489 raft_consensus.cc:515] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.913537 6489 leader_election.cc:290] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [CANDIDATE]: Term 1 election: Requested vote from peers 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203), d7d86a988796472c988683396093ddbb (127.4.183.131:43283)
I20260128 20:50:32.914024 6411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11c3a82ba87f42a7add2b357f095955c" candidate_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d86a988796472c988683396093ddbb"
I20260128 20:50:32.914044 6269 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11c3a82ba87f42a7add2b357f095955c" candidate_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04750e3db41b4677b755e1cf3362917f"
I20260128 20:50:32.914140 6411 raft_consensus.cc:3060] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:32.914215 6269 raft_consensus.cc:3060] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:32.915256 6411 raft_consensus.cc:2468] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1d6cb505304e6b82ea4aa96df1a275 in term 1.
I20260128 20:50:32.915499 6269 raft_consensus.cc:2468] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1d6cb505304e6b82ea4aa96df1a275 in term 1.
I20260128 20:50:32.915683 6059 leader_election.cc:304] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d86a988796472c988683396093ddbb, ff1d6cb505304e6b82ea4aa96df1a275; no voters:
I20260128 20:50:32.915920 6489 raft_consensus.cc:2804] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:32.916181 6489 raft_consensus.cc:697] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 LEADER]: Becoming Leader. State: Replica: ff1d6cb505304e6b82ea4aa96df1a275, State: Running, Role: LEADER
I20260128 20:50:32.916478 6489 consensus_queue.cc:237] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:32.919402 5965 catalog_manager.cc:5654] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129). New cstate: current_term: 1 leader_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } health_report { overall_health: UNKNOWN } } }
W20260128 20:50:32.953364 6316 tablet.cc:2378] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 28 20:50:32 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633430, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:32.997058 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.987348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.4.183.190:39307)
0128 20:50:32.987570 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:32.987580 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:32.987662 (+ 82us) server_negotiation.cc:408] Connection header received
0128 20:50:32.987830 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:32.987833 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:32.987881 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:32.987959 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:32.988888 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.989953 (+ 1065us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.990616 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.990806 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.993576 (+ 2770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:32.993600 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:32.993602 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:32.993627 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:32.995166 (+ 1539us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.995768 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.995773 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.995775 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.995823 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:32.996065 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:32.996070 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:32.996072 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:32.996253 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:32.996359 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:32.996694 (+ 335us) server_negotiation.cc:300] Negotiation successful
0128 20:50:32.996910 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":125}
I20260128 20:50:32.997061 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:32.987204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39307 (local address 127.0.0.1:33432)
0128 20:50:32.987566 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:32.987582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:32.987702 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:32.988010 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:32.988012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:32.988542 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:32.988747 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.988754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.990074 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:32.990080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:32.990492 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:32.990501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:32.990645 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:32.991453 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:32.991477 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:32.993386 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:32.995373 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.995387 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.995394 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.995661 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.995927 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:32.995935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:32.995936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:32.995981 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:32.996379 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:32.996388 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:32.996507 (+ 119us) client_negotiation.cc:769] Sending connection context
0128 20:50:32.996631 (+ 124us) client_negotiation.cc:240] Negotiation successful
0128 20:50:32.996838 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
I20260128 20:50:33.002887 5965 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:33432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260128 20:50:33.003742 5965 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:33.010434 6107 tablet_service.cc:1505] Processing CreateTablet for tablet 113b2943001f45a08c91036677c2b21b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14f28142a1174cbeb9ab4853c262ab3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:50:33.010617 6249 tablet_service.cc:1505] Processing CreateTablet for tablet 113b2943001f45a08c91036677c2b21b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14f28142a1174cbeb9ab4853c262ab3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:50:33.010911 6107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113b2943001f45a08c91036677c2b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:33.010638 6391 tablet_service.cc:1505] Processing CreateTablet for tablet 113b2943001f45a08c91036677c2b21b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14f28142a1174cbeb9ab4853c262ab3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:50:33.010960 6249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113b2943001f45a08c91036677c2b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:33.011226 6391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113b2943001f45a08c91036677c2b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:33.013660 6481 tablet_bootstrap.cc:492] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: Bootstrap starting.
I20260128 20:50:33.013883 6484 tablet_bootstrap.cc:492] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: Bootstrap starting.
I20260128 20:50:33.013883 6482 tablet_bootstrap.cc:492] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: Bootstrap starting.
I20260128 20:50:33.014688 6481 tablet_bootstrap.cc:654] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:33.014919 6484 tablet_bootstrap.cc:654] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:33.015098 6482 tablet_bootstrap.cc:654] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:33.016604 6482 tablet_bootstrap.cc:492] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: No bootstrap required, opened a new log
I20260128 20:50:33.016732 6482 ts_tablet_manager.cc:1403] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260128 20:50:33.016734 6484 tablet_bootstrap.cc:492] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: No bootstrap required, opened a new log
I20260128 20:50:33.016796 6481 tablet_bootstrap.cc:492] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: No bootstrap required, opened a new log
I20260128 20:50:33.016820 6484 ts_tablet_manager.cc:1403] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:50:33.016875 6481 ts_tablet_manager.cc:1403] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260128 20:50:33.017339 6482 raft_consensus.cc:359] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.017372 6484 raft_consensus.cc:359] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.017364 6481 raft_consensus.cc:359] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.017481 6481 raft_consensus.cc:385] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:33.017481 6484 raft_consensus.cc:385] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:33.017481 6482 raft_consensus.cc:385] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:33.017526 6482 raft_consensus.cc:740] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04750e3db41b4677b755e1cf3362917f, State: Initialized, Role: FOLLOWER
I20260128 20:50:33.017527 6484 raft_consensus.cc:740] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1d6cb505304e6b82ea4aa96df1a275, State: Initialized, Role: FOLLOWER
I20260128 20:50:33.017527 6481 raft_consensus.cc:740] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d86a988796472c988683396093ddbb, State: Initialized, Role: FOLLOWER
I20260128 20:50:33.017663 6481 consensus_queue.cc:260] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.017685 6484 consensus_queue.cc:260] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.017802 6482 consensus_queue.cc:260] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.017963 6481 ts_tablet_manager.cc:1434] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 20:50:33.017921 6489 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.018033 6489 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.018078 6488 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.018187 6488 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.018193 6487 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.018291 6487 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.018790 6484 ts_tablet_manager.cc:1434] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260128 20:50:33.018834 6482 ts_tablet_manager.cc:1434] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260128 20:50:33.024420 6487 raft_consensus.cc:493] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:33.024595 6487 raft_consensus.cc:515] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.025065 6487 leader_election.cc:290] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947), 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203)
I20260128 20:50:33.025695 6269 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "113b2943001f45a08c91036677c2b21b" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04750e3db41b4677b755e1cf3362917f" is_pre_election: true
I20260128 20:50:33.025696 6127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "113b2943001f45a08c91036677c2b21b" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" is_pre_election: true
I20260128 20:50:33.025867 6269 raft_consensus.cc:2468] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 0.
I20260128 20:50:33.025867 6127 raft_consensus.cc:2468] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 0.
I20260128 20:50:33.026302 6342 leader_election.cc:304] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d86a988796472c988683396093ddbb, ff1d6cb505304e6b82ea4aa96df1a275; no voters:
I20260128 20:50:33.026510 6487 raft_consensus.cc:2804] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:33.026572 6487 raft_consensus.cc:493] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:33.026612 6487 raft_consensus.cc:3060] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:33.027701 6487 raft_consensus.cc:515] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.028136 6487 leader_election.cc:290] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 election: Requested vote from peers ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947), 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203)
I20260128 20:50:33.028612 6269 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "113b2943001f45a08c91036677c2b21b" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04750e3db41b4677b755e1cf3362917f"
I20260128 20:50:33.028611 6127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "113b2943001f45a08c91036677c2b21b" candidate_uuid: "d7d86a988796472c988683396093ddbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1d6cb505304e6b82ea4aa96df1a275"
I20260128 20:50:33.028735 6269 raft_consensus.cc:3060] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:33.028764 6127 raft_consensus.cc:3060] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:33.029848 6269 raft_consensus.cc:2468] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 1.
I20260128 20:50:33.030287 6342 leader_election.cc:304] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04750e3db41b4677b755e1cf3362917f, d7d86a988796472c988683396093ddbb; no voters:
I20260128 20:50:33.030592 6487 raft_consensus.cc:2804] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:33.030736 6127 raft_consensus.cc:2468] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d86a988796472c988683396093ddbb in term 1.
I20260128 20:50:33.030925 6487 raft_consensus.cc:697] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [term 1 LEADER]: Becoming Leader. State: Replica: d7d86a988796472c988683396093ddbb, State: Running, Role: LEADER
I20260128 20:50:33.031117 6487 consensus_queue.cc:237] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } }
I20260128 20:50:33.032419 6499 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: TxnStatusTablet state changed. Reason: New leader d7d86a988796472c988683396093ddbb. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.032564 6499 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:50:33.032786 6524 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260128 20:50:33.032948 6524 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260128 20:50:33.033612 5967 catalog_manager.cc:5654] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d86a988796472c988683396093ddbb (127.4.183.131). New cstate: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } health_report { overall_health: HEALTHY } } }
I20260128 20:50:33.049401 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.045038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.4.183.131:43283)
0128 20:50:33.045321 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:33.045328 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:33.045350 (+ 22us) server_negotiation.cc:408] Connection header received
0128 20:50:33.045579 (+ 229us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:33.045582 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:33.045650 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:33.045736 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:33.046241 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.047042 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.048110 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.048594 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.048662 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.048774 (+ 112us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.048864 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:33.049055 (+ 191us) server_negotiation.cc:300] Negotiation successful
0128 20:50:33.049184 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260128 20:50:33.049903 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.045000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:43283 (local address 127.0.0.1:55100)
0128 20:50:33.045282 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:33.045299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:33.045450 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:33.045788 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:33.045792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:33.045816 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:33.046097 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:33.046106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.047297 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.047301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:33.047971 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:33.047982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.048139 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.048207 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.048920 (+ 713us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.048944 (+ 24us) client_negotiation.cc:769] Sending connection context
0128 20:50:33.049602 (+ 658us) client_negotiation.cc:240] Negotiation successful
0128 20:50:33.049722 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260128 20:50:33.054486 6269 raft_consensus.cc:1275] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Refusing update from remote peer d7d86a988796472c988683396093ddbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:33.054498 6127 raft_consensus.cc:1275] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Refusing update from remote peer d7d86a988796472c988683396093ddbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:33.055078 6499 consensus_queue.cc:1048] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:33.055315 6487 consensus_queue.cc:1048] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:33.057235 6489 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: TxnStatusTablet state changed. Reason: New leader d7d86a988796472c988683396093ddbb. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.057358 6489 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.057082 6488 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: TxnStatusTablet state changed. Reason: New leader d7d86a988796472c988683396093ddbb. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.058496 6488 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.059701 6499 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.059842 6499 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:50:33.060333 6489 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.060479 6489 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P ff1d6cb505304e6b82ea4aa96df1a275: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.061898 6491 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.062016 6491 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P d7d86a988796472c988683396093ddbb: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:50:33.063846 6488 tablet_replica.cc:442] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d86a988796472c988683396093ddbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1d6cb505304e6b82ea4aa96df1a275" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 33947 } } peers { permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 } } peers { permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 } } }
I20260128 20:50:33.063988 6488 tablet_replica.cc:445] T 113b2943001f45a08c91036677c2b21b P 04750e3db41b4677b755e1cf3362917f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:33.072126 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.068086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.4.183.129:33947)
0128 20:50:33.068261 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:33.068266 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:33.068379 (+ 113us) server_negotiation.cc:408] Connection header received
0128 20:50:33.068558 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:33.068562 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:33.068639 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:33.068752 (+ 113us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:33.069072 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.069877 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.070654 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.070861 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.070931 (+ 70us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.071608 (+ 677us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.071741 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:33.071873 (+ 132us) server_negotiation.cc:300] Negotiation successful
0128 20:50:33.071970 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260128 20:50:33.085182 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.081820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:47933 (local address 127.4.183.131:43283)
0128 20:50:33.082008 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:33.082013 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:33.082209 (+ 196us) server_negotiation.cc:408] Connection header received
0128 20:50:33.082410 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:33.082417 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:33.082467 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:33.082549 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:33.082982 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.083477 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.084363 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.084571 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.084614 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.084694 (+ 80us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.084782 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:33.084935 (+ 153us) server_negotiation.cc:300] Negotiation successful
0128 20:50:33.085005 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260128 20:50:33.085119 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.081701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:43283 (local address 127.4.183.129:47933)
0128 20:50:33.082057 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:33.082071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:33.082289 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:33.082573 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:33.082576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:33.082584 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:33.082855 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:33.082862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.083601 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.083605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:33.084189 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:33.084201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.084360 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.084425 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.084790 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.084798 (+ 8us) client_negotiation.cc:769] Sending connection context
0128 20:50:33.084874 (+ 76us) client_negotiation.cc:240] Negotiation successful
0128 20:50:33.084968 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":123,"threads_started":1}
I20260128 20:50:33.095199 6269 raft_consensus.cc:1275] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Refusing update from remote peer ff1d6cb505304e6b82ea4aa96df1a275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:33.095652 6411 raft_consensus.cc:1275] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Refusing update from remote peer ff1d6cb505304e6b82ea4aa96df1a275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:33.095880 6489 consensus_queue.cc:1048] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04750e3db41b4677b755e1cf3362917f" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:33.096422 6489 consensus_queue.cc:1048] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d86a988796472c988683396093ddbb" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:33.101013 6496 mvcc.cc:204] Tried to move back new op lower bound from 7248418541951815680 to 7248418541229465600. Current Snapshot: MvccSnapshot[applied={T|T < 7248418541951815680}]
I20260128 20:50:33.125306 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.121648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:53629 (local address 127.4.183.129:33947)
0128 20:50:33.121864 (+ 216us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:33.121869 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:33.121924 (+ 55us) server_negotiation.cc:408] Connection header received
0128 20:50:33.122123 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:33.122128 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:33.122209 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:33.122352 (+ 143us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:33.122959 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.123576 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.124267 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.124470 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.124523 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.124671 (+ 148us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.124757 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:33.125018 (+ 261us) server_negotiation.cc:300] Negotiation successful
0128 20:50:33.125079 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260128 20:50:33.125418 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.121465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:33947 (local address 127.4.183.131:53629)
0128 20:50:33.121838 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:33.121854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:33.121964 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:33.122393 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:33.122399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:33.122412 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:33.122774 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:33.122782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.123715 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.123719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:33.124142 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:33.124149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.124285 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.124351 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:33.124806 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:33.124815 (+ 9us) client_negotiation.cc:769] Sending connection context
0128 20:50:33.124935 (+ 120us) client_negotiation.cc:240] Negotiation successful
0128 20:50:33.125225 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":129,"threads_started":1}
I20260128 20:50:33.160682 5966 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:33.160881 5966 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:33.163686 5966 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4bb6c592c8d84650b44e7cb4187e6cf9: Sending DeleteTablet for 3 replicas of tablet 11c3a82ba87f42a7add2b357f095955c
I20260128 20:50:33.164605 6391 tablet_service.cc:1552] Processing DeleteTablet for tablet 11c3a82ba87f42a7add2b357f095955c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:33 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:55086
I20260128 20:50:33.164851 6504 tablet_replica.cc:333] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: stopping tablet replica
I20260128 20:50:33.164997 6504 raft_consensus.cc:2243] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:33.165177 6504 raft_consensus.cc:2272] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:33.165998 6249 tablet_service.cc:1552] Processing DeleteTablet for tablet 11c3a82ba87f42a7add2b357f095955c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:33 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:50168
I20260128 20:50:33.166214 6504 ts_tablet_manager.cc:1916] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:33.166383 6107 tablet_service.cc:1552] Processing DeleteTablet for tablet 11c3a82ba87f42a7add2b357f095955c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:33 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:51840
I20260128 20:50:33.166595 6506 tablet_replica.cc:333] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: stopping tablet replica
I20260128 20:50:33.166712 6506 raft_consensus.cc:2243] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:50:33.167055 6506 raft_consensus.cc:2272] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:33.169308 6505 tablet_replica.cc:333] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: stopping tablet replica
I20260128 20:50:33.169490 6505 raft_consensus.cc:2243] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:33.169648 6505 raft_consensus.cc:2272] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:33.170643 6506 ts_tablet_manager.cc:1916] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:33.174181 6505 ts_tablet_manager.cc:1916] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jan 28 20:50:33 dist-test-slave-hss7 krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633432, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:33.177860 6505 ts_tablet_manager.cc:1929] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:33.177989 6505 log.cc:1199] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/11c3a82ba87f42a7add2b357f095955c
I20260128 20:50:33.178297 6505 ts_tablet_manager.cc:1950] T 11c3a82ba87f42a7add2b357f095955c P 04750e3db41b4677b755e1cf3362917f: Deleting consensus metadata
I20260128 20:50:33.179289 5951 catalog_manager.cc:4985] TS 04750e3db41b4677b755e1cf3362917f (127.4.183.130:38203): tablet 11c3a82ba87f42a7add2b357f095955c (table test-table [id=91f97408b6084f0c9a35de69fb3be2d6]) successfully deleted
I20260128 20:50:33.180284 6506 ts_tablet_manager.cc:1929] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:33.180393 6506 log.cc:1199] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/11c3a82ba87f42a7add2b357f095955c
I20260128 20:50:33.180696 6506 ts_tablet_manager.cc:1950] T 11c3a82ba87f42a7add2b357f095955c P ff1d6cb505304e6b82ea4aa96df1a275: Deleting consensus metadata
I20260128 20:50:33.181188 6504 ts_tablet_manager.cc:1929] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:33.181304 6504 log.cc:1199] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/11c3a82ba87f42a7add2b357f095955c
I20260128 20:50:33.181609 6504 ts_tablet_manager.cc:1950] T 11c3a82ba87f42a7add2b357f095955c P d7d86a988796472c988683396093ddbb: Deleting consensus metadata
I20260128 20:50:33.181850 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.167603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.4.183.129:33947)
0128 20:50:33.167738 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:33.167743 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:33.168354 (+ 611us) server_negotiation.cc:408] Connection header received
0128 20:50:33.168564 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:33.168568 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:33.168625 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:33.168682 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:33.170562 (+ 1880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.171321 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.172317 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.172536 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.176505 (+ 3969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:33.176546 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:33.176563 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:33.176614 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:33.179093 (+ 2479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:33.179671 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:33.179679 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:33.179681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:33.179775 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:33.180164 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:33.180169 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:33.180174 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:33.180459 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:33.180679 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:33.181442 (+ 763us) server_negotiation.cc:300] Negotiation successful
0128 20:50:33.181627 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 20:50:33.182586 5953 catalog_manager.cc:4985] TS d7d86a988796472c988683396093ddbb (127.4.183.131:43283): tablet 11c3a82ba87f42a7add2b357f095955c (table test-table [id=91f97408b6084f0c9a35de69fb3be2d6]) successfully deleted
W20260128 20:50:33.183298 6167 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862
I20260128 20:50:33.183537 5951 catalog_manager.cc:4985] TS ff1d6cb505304e6b82ea4aa96df1a275 (127.4.183.129:33947): tablet 11c3a82ba87f42a7add2b357f095955c (table test-table [id=91f97408b6084f0c9a35de69fb3be2d6]) successfully deleted
I20260128 20:50:33.196542 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:33.186241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.4.183.190:39307)
0128 20:50:33.186386 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:33.186390 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:33.186845 (+ 455us) server_negotiation.cc:408] Connection header received
0128 20:50:33.187024 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:33.187028 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:33.187081 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:33.187132 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:33.188535 (+ 1403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.189432 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:33.190392 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:33.190600 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:33.192504 (+ 1904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:33.192531 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:33.192535 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:33.192575 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:33.194526 (+ 1951us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:33.195148 (+ 622us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:33.195153 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:33.195155 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:33.195215 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:33.195491 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:33.195496 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:33.195501 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:33.195729 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:33.195826 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:33.196192 (+ 366us) server_negotiation.cc:300] Negotiation successful
0128 20:50:33.196322 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260128 20:50:33.197299 5966 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33436
I20260128 20:50:33.199047 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6041
I20260128 20:50:33.210752 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6184
I20260128 20:50:33.219805 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6326
I20260128 20:50:33.227748 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 5935
2026-01-28T20:50:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5533 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:33 dist-test-slave-hss7 krb5kdc[6545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:33 dist-test-slave-hss7 krb5kdc[6545](info): set up 2 sockets
Jan 28 20:50:33 dist-test-slave-hss7 krb5kdc[6545](info): commencing operation
krb5kdc: starting...
W20260128 20:50:35.286664 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.032s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:35 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:35.445611 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:33587
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:35.566057 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:35.566494 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:35.566615 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:35.571166 6561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:35.571326 6561 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260128 20:50:35.571360 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:35.571414 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:35.571477 6561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:35.571521 6561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:35.576864 6561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:33587
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:35.578258 6561 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:35.579268 6561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:35.585690 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:35.585706 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:35.585695 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:35.586417 6561 server_base.cc:1060] running on GCE node
I20260128 20:50:35.587042 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:35.588163 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:35.589376 6561 hybrid_clock.cc:648] HybridClock initialized: now 1769633435589344 us; error 60 us; skew 500 ppm
Jan 28 20:50:35 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633435, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:35.592917 6561 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:35.594394 6561 webserver.cc:492] Webserver started at http://127.4.183.190:36147/ using document root <none> and password file <none>
I20260128 20:50:35.595038 6561 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:35.595129 6561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:35.595361 6561 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:35.597487 6561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "0ad5af48abd2405a950984ce55300382"
format_stamp: "Formatted at 2026-01-28 20:50:35 on dist-test-slave-hss7"
server_key: "a6318f33ec022706771cc472caed89f8"
server_key_iv: "8b35f7856a0f42c3d0e2f18a1405cdd3"
server_key_version: "encryptionkey@0"
I20260128 20:50:35.598037 6561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "0ad5af48abd2405a950984ce55300382"
format_stamp: "Formatted at 2026-01-28 20:50:35 on dist-test-slave-hss7"
server_key: "a6318f33ec022706771cc472caed89f8"
server_key_iv: "8b35f7856a0f42c3d0e2f18a1405cdd3"
server_key_version: "encryptionkey@0"
I20260128 20:50:35.602509 6561 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:35.605415 6576 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:35.606700 6561 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:35.606907 6561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "0ad5af48abd2405a950984ce55300382"
format_stamp: "Formatted at 2026-01-28 20:50:35 on dist-test-slave-hss7"
server_key: "a6318f33ec022706771cc472caed89f8"
server_key_iv: "8b35f7856a0f42c3d0e2f18a1405cdd3"
server_key_version: "encryptionkey@0"
I20260128 20:50:35.607049 6561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:35.629611 6561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:35.633865 6561 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:35.634142 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:35.643719 6561 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:33587
I20260128 20:50:35.643719 6638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:33587 every 8 connection(s)
I20260128 20:50:35.644912 6561 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:35.648792 6639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:35.653617 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6561
I20260128 20:50:35.653755 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:35.653990 4830 external_mini_cluster.cc:1468] Setting key 8c1ba519c6280d2c5d36ee58e0c7a3d2
I20260128 20:50:35.655179 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Bootstrap starting.
I20260128 20:50:35.657800 6639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:35.658630 6639 log.cc:826] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:35.660923 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: No bootstrap required, opened a new log
Jan 28 20:50:35 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:35.664352 6639 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5af48abd2405a950984ce55300382" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33587 } }
I20260128 20:50:35.664664 6639 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:35.664739 6639 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad5af48abd2405a950984ce55300382, State: Initialized, Role: FOLLOWER
I20260128 20:50:35.665360 6639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5af48abd2405a950984ce55300382" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33587 } }
I20260128 20:50:35.665527 6639 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:35.665573 6639 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:35.665710 6639 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:35.667053 6639 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5af48abd2405a950984ce55300382" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33587 } }
I20260128 20:50:35.667516 6639 leader_election.cc:304] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ad5af48abd2405a950984ce55300382; no voters:
I20260128 20:50:35.667913 6639 leader_election.cc:290] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:35.668227 6644 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:35.668504 6644 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad5af48abd2405a950984ce55300382, State: Running, Role: LEADER
I20260128 20:50:35.668926 6644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5af48abd2405a950984ce55300382" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33587 } }
I20260128 20:50:35.669431 6639 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:35.670452 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:35.655339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.4.183.190:33587)
0128 20:50:35.655828 (+ 489us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:35.655838 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:35.655878 (+ 40us) server_negotiation.cc:408] Connection header received
0128 20:50:35.656532 (+ 654us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:35.656553 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:35.656914 (+ 361us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:35.657488 (+ 574us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:35.658506 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.659331 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:35.660065 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.660394 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:35.663555 (+ 3161us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:35.663586 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:35.663604 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:35.663674 (+ 70us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:35.667080 (+ 3406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:35.667753 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:35.667760 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:35.667767 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:35.667889 (+ 122us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:35.668186 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:35.668190 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:35.668192 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:35.668693 (+ 501us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:35.668915 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:35.669296 (+ 381us) server_negotiation.cc:300] Negotiation successful
0128 20:50:35.669607 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260128 20:50:35.670818 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ad5af48abd2405a950984ce55300382. Latest consensus state: current_term: 1 leader_uuid: "0ad5af48abd2405a950984ce55300382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5af48abd2405a950984ce55300382" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33587 } } }
I20260128 20:50:35.670931 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:35.671275 6645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ad5af48abd2405a950984ce55300382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5af48abd2405a950984ce55300382" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33587 } } }
I20260128 20:50:35.671358 6645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382 [sys.catalog]: This master's current role is: LEADER
W20260128 20:50:35.674404 6659 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:50:35.674531 6659 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:50:35.674628 6653 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:35.675467 6653 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:35.681879 6653 catalog_manager.cc:1357] Generated new cluster ID: 1cc310fe1df442b2b99954d0b8973297
I20260128 20:50:35.681985 6653 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:35.695833 6653 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:35.697868 6653 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:35.708698 6653 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Generated new TSK 0
I20260128 20:50:35.710101 6653 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:35.774212 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:33587
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:35.895493 6667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:35.895769 6667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:35.895882 6667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:35.899749 6667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:35.899842 6667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:35.899902 6667 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260128 20:50:35.899952 6667 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:50:35.900048 6667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:35.904809 6667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:35.906147 6667 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:35.907119 6667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:35.914973 6673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:35.914963 6672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:35.915325 6667 server_base.cc:1060] running on GCE node
W20260128 20:50:35.914963 6675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:35.915896 6667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:35.916720 6667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:35.917946 6667 hybrid_clock.cc:648] HybridClock initialized: now 1769633435917922 us; error 46 us; skew 500 ppm
Jan 28 20:50:35 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633435, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:35.921216 6667 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:35.922472 6667 webserver.cc:492] Webserver started at http://127.4.183.129:40815/ using document root <none> and password file <none>
I20260128 20:50:35.923032 6667 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:35.923084 6667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:35.923255 6667 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:35.925078 6667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "4b934f8638a849d99e8019d77f5d406d"
format_stamp: "Formatted at 2026-01-28 20:50:35 on dist-test-slave-hss7"
server_key: "ee94910026b25baa2e1b897d8b1d2603"
server_key_iv: "90d6138dd26dadc26290335937b61477"
server_key_version: "encryptionkey@0"
I20260128 20:50:35.925657 6667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "4b934f8638a849d99e8019d77f5d406d"
format_stamp: "Formatted at 2026-01-28 20:50:35 on dist-test-slave-hss7"
server_key: "ee94910026b25baa2e1b897d8b1d2603"
server_key_iv: "90d6138dd26dadc26290335937b61477"
server_key_version: "encryptionkey@0"
I20260128 20:50:35.929806 6667 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:35.932575 6682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:35.934157 6667 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:35.934351 6667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "4b934f8638a849d99e8019d77f5d406d"
format_stamp: "Formatted at 2026-01-28 20:50:35 on dist-test-slave-hss7"
server_key: "ee94910026b25baa2e1b897d8b1d2603"
server_key_iv: "90d6138dd26dadc26290335937b61477"
server_key_version: "encryptionkey@0"
I20260128 20:50:35.934461 6667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:35.954548 6667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:35.958333 6667 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:35.958516 6667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:35.960008 6667 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:35.960122 6667 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:35.960201 6667 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:35.960248 6667 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:50:35 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633435, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:35.972348 6667 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:44279
I20260128 20:50:35.972465 6797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:44279 every 8 connection(s)
I20260128 20:50:35.973608 6667 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:50:35.976301 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:35.960940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:42445 (local address 127.4.183.190:33587)
0128 20:50:35.961110 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:35.961123 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:35.962039 (+ 916us) server_negotiation.cc:408] Connection header received
0128 20:50:35.963951 (+ 1912us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:35.963955 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:35.964021 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:35.964107 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:35.966323 (+ 2216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.967207 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:35.968313 (+ 1106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.968547 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:35.971767 (+ 3220us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:35.971794 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:35.971796 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:35.971874 (+ 78us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:35.973614 (+ 1740us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:35.974314 (+ 700us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:35.974318 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:35.974320 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:35.974375 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:35.974814 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:35.974817 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:35.974819 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:35.975010 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:35.975098 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:35.976020 (+ 922us) server_negotiation.cc:300] Negotiation successful
0128 20:50:35.976163 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260128 20:50:35.977123 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:35.961296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.4.183.129:42445)
0128 20:50:35.961844 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:35.961891 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:35.963693 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:35.964363 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:35.964379 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:35.965061 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:35.966051 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:35.966077 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.967397 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:35.967401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:35.968134 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:35.968148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.968487 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:35.969314 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:35.969351 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:35.971536 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:35.973804 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:35.973809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:35.973819 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:35.974178 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:35.974543 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:35.974547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:35.974549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:35.974672 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:35.975238 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:35.975256 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:35.975650 (+ 394us) client_negotiation.cc:769] Sending connection context
0128 20:50:35.975948 (+ 298us) client_negotiation.cc:240] Negotiation successful
0128 20:50:35.976349 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":158,"threads_started":1}
I20260128 20:50:35.978854 6798 heartbeater.cc:344] Connected to a master server at 127.4.183.190:33587
I20260128 20:50:35.979086 6798 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:35.979804 6798 heartbeater.cc:507] Master 127.4.183.190:33587 requested a full tablet report, sending...
I20260128 20:50:35.981647 6592 ts_manager.cc:194] Registered new tserver with Master: 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279)
I20260128 20:50:35.981937 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6667
I20260128 20:50:35.982072 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:35.982318 4830 external_mini_cluster.cc:1468] Setting key c4bebb2a0c9871800431a357a1370c29
I20260128 20:50:35.983249 6592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:42445
I20260128 20:50:35.989495 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:35.981331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.4.183.129:56435)
0128 20:50:35.981646 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:35.981659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:35.981790 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:35.982049 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:35.982052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:35.982266 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:35.982497 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:35.982503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.983435 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:35.983439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:35.983999 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:35.984010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.984164 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:35.984908 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:35.984936 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:35.985410 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:35.987696 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:35.987699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:35.987701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:35.987897 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:35.988250 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:35.988253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:35.988255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:35.988310 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:35.988801 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:35.988804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:35.988908 (+ 104us) client_negotiation.cc:769] Sending connection context
0128 20:50:35.989009 (+ 101us) client_negotiation.cc:240] Negotiation successful
0128 20:50:35.989314 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":138,"threads_started":1}
I20260128 20:50:35.989547 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:35.981345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:56435 (local address 127.4.183.190:33587)
0128 20:50:35.981517 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:35.981522 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:35.981738 (+ 216us) server_negotiation.cc:408] Connection header received
0128 20:50:35.981888 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:35.981892 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:35.981943 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:35.982029 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:35.982625 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.983291 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:35.984163 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:35.984381 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:35.985588 (+ 1207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:35.985611 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:35.985617 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:35.985650 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:35.987536 (+ 1886us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:35.988014 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:35.988020 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:35.988023 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:35.988087 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:35.988430 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:35.988436 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:35.988441 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:35.988670 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:35.988792 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:35.989103 (+ 311us) server_negotiation.cc:300] Negotiation successful
0128 20:50:35.989283 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:36.041095 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:33587
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:36.162667 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:36.162945 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:36.163013 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:36.166561 6809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:36.166663 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:36.166793 6809 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260128 20:50:36.166841 6809 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:50:36.166910 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:36.172034 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:36.173322 6809 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:36.174214 6809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:36.181910 6814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:36.181910 6817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:36.181912 6815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:36.182267 6809 server_base.cc:1060] running on GCE node
I20260128 20:50:36.182706 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:36.183329 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:36.184561 6809 hybrid_clock.cc:648] HybridClock initialized: now 1769633436184539 us; error 36 us; skew 500 ppm
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633436, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:36.188007 6809 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:36.189638 6809 webserver.cc:492] Webserver started at http://127.4.183.130:37859/ using document root <none> and password file <none>
I20260128 20:50:36.190326 6809 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:36.190429 6809 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:36.190680 6809 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:36.192715 6809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "36252231303540e2bad76a9e9696d0c1"
format_stamp: "Formatted at 2026-01-28 20:50:36 on dist-test-slave-hss7"
server_key: "97e5dc29f5572998a63c81b2759e58d4"
server_key_iv: "8e1397f91ae9b78e29c17eba1d68c90d"
server_key_version: "encryptionkey@0"
I20260128 20:50:36.193387 6809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "36252231303540e2bad76a9e9696d0c1"
format_stamp: "Formatted at 2026-01-28 20:50:36 on dist-test-slave-hss7"
server_key: "97e5dc29f5572998a63c81b2759e58d4"
server_key_iv: "8e1397f91ae9b78e29c17eba1d68c90d"
server_key_version: "encryptionkey@0"
I20260128 20:50:36.197636 6809 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:50:36.200606 6824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:36.202076 6809 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:36.202301 6809 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "36252231303540e2bad76a9e9696d0c1"
format_stamp: "Formatted at 2026-01-28 20:50:36 on dist-test-slave-hss7"
server_key: "97e5dc29f5572998a63c81b2759e58d4"
server_key_iv: "8e1397f91ae9b78e29c17eba1d68c90d"
server_key_version: "encryptionkey@0"
I20260128 20:50:36.202450 6809 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:36.213596 6809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:36.217978 6809 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:36.218346 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:36.220891 6809 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:36.221002 6809 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:36.221176 6809 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:36.221216 6809 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633436, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:36.236533 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.221861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:48757 (local address 127.4.183.190:33587)
0128 20:50:36.222072 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.222078 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.223294 (+ 1216us) server_negotiation.cc:408] Connection header received
0128 20:50:36.224654 (+ 1360us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.224658 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.224717 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.224805 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:36.226944 (+ 2139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.227627 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.228678 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.228909 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.232453 (+ 3544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:36.232480 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:36.232482 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:36.232520 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:36.234236 (+ 1716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.234888 (+ 652us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.234892 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.234894 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.234949 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.235328 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.235334 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.235338 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.235522 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:36.235636 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.236203 (+ 567us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.236359 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260128 20:50:36.237015 6809 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:46683
I20260128 20:50:36.237411 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.222236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.4.183.130:48757)
0128 20:50:36.222912 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.222980 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.224373 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.225047 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.225062 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.225789 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:36.226706 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.226724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.227802 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.227809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.228511 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.228524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.228837 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.229714 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:36.229775 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:36.232204 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:36.234412 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.234421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.234439 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.234761 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.235048 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.235051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.235055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.235224 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.235636 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:36.235647 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:36.235931 (+ 284us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.236161 (+ 230us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.236478 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":153,"threads_started":1}
I20260128 20:50:36.237061 6939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:46683 every 8 connection(s)
I20260128 20:50:36.238654 6809 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:36.240623 6940 heartbeater.cc:344] Connected to a master server at 127.4.183.190:33587
I20260128 20:50:36.240885 6940 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:36.241698 6940 heartbeater.cc:507] Master 127.4.183.190:33587 requested a full tablet report, sending...
I20260128 20:50:36.243007 6592 ts_manager.cc:194] Registered new tserver with Master: 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683)
I20260128 20:50:36.243927 6592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:48757
I20260128 20:50:36.248584 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6809
I20260128 20:50:36.248756 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:36.249022 4830 external_mini_cluster.cc:1468] Setting key bdcff603df7d03b28c16ab985fb472fe
I20260128 20:50:36.251255 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.241889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:46435 (local address 127.4.183.190:33587)
0128 20:50:36.242039 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.242044 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.242335 (+ 291us) server_negotiation.cc:408] Connection header received
0128 20:50:36.242408 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.242414 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.242482 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.242560 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:36.243344 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.244133 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.245569 (+ 1436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.245749 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.247122 (+ 1373us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:36.247149 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:36.247151 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:36.247185 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:36.248929 (+ 1744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.249692 (+ 763us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.249700 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.249702 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.249805 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.250152 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.250159 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.250161 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.250417 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:36.250555 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.250857 (+ 302us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.251027 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260128 20:50:36.251292 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.241834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.4.183.130:46435)
0128 20:50:36.242139 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.242156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.242282 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.242646 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.242650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.242906 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:36.243198 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.243207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.244530 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.244539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.245332 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.245351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.245560 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.246372 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:36.246392 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:36.246904 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:36.249102 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.249109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.249111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.249475 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.249930 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.249935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.249938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.250012 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.250537 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:36.250541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:36.250668 (+ 127us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.250790 (+ 122us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.251027 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:36.308481 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:33587
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:36.436487 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:36.436825 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:36.436949 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:36.441641 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:36.441772 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:36.441867 6951 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260128 20:50:36.441917 6951 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:50:36.442005 6951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:36.446971 6951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:36.448354 6951 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:36.449432 6951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:36.457501 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:36.457594 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:36.457501 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:36.457788 6951 server_base.cc:1060] running on GCE node
I20260128 20:50:36.458268 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:36.459474 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:36.462929 6951 hybrid_clock.cc:648] HybridClock initialized: now 1769633436462856 us; error 135 us; skew 500 ppm
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633436, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:36.466440 6951 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:36.467739 6951 webserver.cc:492] Webserver started at http://127.4.183.131:36189/ using document root <none> and password file <none>
I20260128 20:50:36.468348 6951 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:36.468405 6951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:36.468577 6951 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:36.471271 6951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "506914e417bf4007985e767b4ded9e03"
format_stamp: "Formatted at 2026-01-28 20:50:36 on dist-test-slave-hss7"
server_key: "66961cacd3e9c3d172f63db305a7c77d"
server_key_iv: "bd5460790122b98af0d0a937e6061066"
server_key_version: "encryptionkey@0"
I20260128 20:50:36.471872 6951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "506914e417bf4007985e767b4ded9e03"
format_stamp: "Formatted at 2026-01-28 20:50:36 on dist-test-slave-hss7"
server_key: "66961cacd3e9c3d172f63db305a7c77d"
server_key_iv: "bd5460790122b98af0d0a937e6061066"
server_key_version: "encryptionkey@0"
I20260128 20:50:36.476715 6951 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260128 20:50:36.481055 6966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:36.482586 6951 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:36.482750 6951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "506914e417bf4007985e767b4ded9e03"
format_stamp: "Formatted at 2026-01-28 20:50:36 on dist-test-slave-hss7"
server_key: "66961cacd3e9c3d172f63db305a7c77d"
server_key_iv: "bd5460790122b98af0d0a937e6061066"
server_key_version: "encryptionkey@0"
I20260128 20:50:36.482867 6951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:36.511624 6951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:36.515247 6951 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:36.515457 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:36.517427 6951 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:36.517534 6951 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:36.517632 6951 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:36.517685 6951 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633436, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:36.530555 6951 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:44061
I20260128 20:50:36.530896 7081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:44061 every 8 connection(s)
I20260128 20:50:36.531697 6951 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:36.533707 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.518300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:38847 (local address 127.4.183.190:33587)
0128 20:50:36.518584 (+ 284us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.518591 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.520103 (+ 1512us) server_negotiation.cc:408] Connection header received
0128 20:50:36.521557 (+ 1454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.521562 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.521642 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.521749 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:36.524348 (+ 2599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.525252 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.526281 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.526505 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.529648 (+ 3143us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:36.529680 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:36.529683 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:36.529715 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:36.531425 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.531995 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.531999 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.532003 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.532056 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.532430 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.532434 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.532436 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.532663 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:36.532770 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.533375 (+ 605us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.533518 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 20:50:36.534670 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.518986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.4.183.131:38847)
0128 20:50:36.519846 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.519922 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.521182 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.522258 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.522279 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.523100 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:36.524092 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.524116 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.525408 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.525414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.526119 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.526133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.526428 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.527111 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:36.527167 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:36.529464 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:36.531562 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.531568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.531581 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.531887 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.532166 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.532169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.532172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.532297 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.532783 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:36.532796 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:36.533024 (+ 228us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.533330 (+ 306us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.533625 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":155,"threads_started":1}
I20260128 20:50:36.536387 7082 heartbeater.cc:344] Connected to a master server at 127.4.183.190:33587
I20260128 20:50:36.536975 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 6951
I20260128 20:50:36.537177 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:36.537518 4830 external_mini_cluster.cc:1468] Setting key 4cbc3686f9c3e9fb58dc17992f8ded57
I20260128 20:50:36.537881 7082 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:36.538754 7082 heartbeater.cc:507] Master 127.4.183.190:33587 requested a full tablet report, sending...
I20260128 20:50:36.540134 6592 ts_manager.cc:194] Registered new tserver with Master: 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:36.541041 6592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:38847
I20260128 20:50:36.550109 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.541573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:36751 (local address 127.4.183.190:33587)
0128 20:50:36.541713 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.541717 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.541946 (+ 229us) server_negotiation.cc:408] Connection header received
0128 20:50:36.542198 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.542215 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.542301 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.542380 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:36.543844 (+ 1464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.544789 (+ 945us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.545720 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.545906 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.546839 (+ 933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:36.546858 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:36.546860 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:36.546886 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:36.548397 (+ 1511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.548871 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.548875 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.548877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.548927 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.549277 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.549280 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.549281 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.549475 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:36.549565 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.549876 (+ 311us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.549988 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260128 20:50:36.550125 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.541533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.4.183.131:36751)
0128 20:50:36.541830 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.541866 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.542035 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.542668 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.542672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.543118 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:36.543590 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.543609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.545001 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.545006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.545588 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.545598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.545710 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.546309 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:36.546323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:36.546697 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:36.548545 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.548549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.548550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.548746 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.549052 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.549055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.549056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.549111 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.549603 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:36.549606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:36.549715 (+ 109us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.549818 (+ 103us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.549952 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20260128 20:50:36.554008 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:36.585068 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.574729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.4.183.190:33587)
0128 20:50:36.574930 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.574934 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.574947 (+ 13us) server_negotiation.cc:408] Connection header received
0128 20:50:36.574988 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.574991 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.575035 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.575139 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:36.576103 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.576629 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.577470 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.577689 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.580768 (+ 3079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:36.580799 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:36.580801 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:36.580897 (+ 96us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:36.582863 (+ 1966us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.583515 (+ 652us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.583519 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.583521 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.583593 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.583849 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.583851 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.583852 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.584033 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:36.584161 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.584712 (+ 551us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.584860 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260128 20:50:36.588929 6592 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:50:36.591657 6592 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:36.608366 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.604029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:44279 (local address 127.0.0.1:43378)
0128 20:50:36.604820 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.604833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.604908 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.605379 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.605384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.605407 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:36.605731 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.605739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.606830 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.606835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.607837 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.607845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.607957 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.607976 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.608044 (+ 68us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.608150 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":104,"threads_started":1}
I20260128 20:50:36.608366 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.603226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:46683 (local address 127.0.0.1:53562)
0128 20:50:36.604071 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.604116 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.604267 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.604893 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.604899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.604921 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:36.605285 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.605293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.606791 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.606795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.607721 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.607733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.607849 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.607940 (+ 91us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.608052 (+ 112us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.608155 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":109,"threads_started":1}
I20260128 20:50:36.609061 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.604177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.4.183.129:44279)
0128 20:50:36.604888 (+ 711us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.604895 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.604919 (+ 24us) server_negotiation.cc:408] Connection header received
0128 20:50:36.605034 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.605040 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.605240 (+ 200us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.605379 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:36.605866 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.606678 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.607994 (+ 1316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.608584 (+ 590us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.608727 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.608807 (+ 80us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.608922 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":96,"threads_started":1}
I20260128 20:50:36.609236 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.603329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.4.183.130:46683)
0128 20:50:36.603685 (+ 356us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.603692 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.604402 (+ 710us) server_negotiation.cc:408] Connection header received
0128 20:50:36.604512 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.604518 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.604725 (+ 207us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.604891 (+ 166us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:36.605444 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.606644 (+ 1200us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.608265 (+ 1621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.608799 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.608892 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.608952 (+ 60us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.609034 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":150,"threads_started":1}
I20260128 20:50:36.609458 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.603243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:44061 (local address 127.0.0.1:49638)
0128 20:50:36.604530 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.604547 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.604648 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.605628 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.605631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.605651 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:36.605916 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.605927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.607654 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.607658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.608792 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.608802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.608914 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.608934 (+ 20us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.608988 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.609063 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":101,"threads_started":1}
I20260128 20:50:36.610760 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.604489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.4.183.131:44061)
0128 20:50:36.605199 (+ 710us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.605205 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.605229 (+ 24us) server_negotiation.cc:408] Connection header received
0128 20:50:36.605310 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.605316 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.605480 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.605641 (+ 161us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:36.606057 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.607234 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.609602 (+ 2368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.610226 (+ 624us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.610392 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.610477 (+ 85us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.610587 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":102,"threads_started":1}
I20260128 20:50:36.612550 6874 tablet_service.cc:1505] Processing CreateTablet for tablet 2b4a82f6a359489097cd65f38ab6e104 (DEFAULT_TABLE table=test-table [id=28fd3b10c0f7413ba1e3fb09df0aafaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:36.612547 6732 tablet_service.cc:1505] Processing CreateTablet for tablet 2b4a82f6a359489097cd65f38ab6e104 (DEFAULT_TABLE table=test-table [id=28fd3b10c0f7413ba1e3fb09df0aafaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:36.613776 6732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4a82f6a359489097cd65f38ab6e104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:36.613776 6874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4a82f6a359489097cd65f38ab6e104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:36.614037 7016 tablet_service.cc:1505] Processing CreateTablet for tablet 2b4a82f6a359489097cd65f38ab6e104 (DEFAULT_TABLE table=test-table [id=28fd3b10c0f7413ba1e3fb09df0aafaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:36.615197 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4a82f6a359489097cd65f38ab6e104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:36.621901 7105 tablet_bootstrap.cc:492] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Bootstrap starting.
I20260128 20:50:36.622689 7106 tablet_bootstrap.cc:492] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Bootstrap starting.
I20260128 20:50:36.622814 7107 tablet_bootstrap.cc:492] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Bootstrap starting.
I20260128 20:50:36.624986 7105 tablet_bootstrap.cc:654] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:36.625007 7106 tablet_bootstrap.cc:654] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:36.625015 7107 tablet_bootstrap.cc:654] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:36.626010 7105 log.cc:826] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:36.626502 7107 log.cc:826] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:36.626502 7106 log.cc:826] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:36.628749 7106 tablet_bootstrap.cc:492] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: No bootstrap required, opened a new log
I20260128 20:50:36.628770 7107 tablet_bootstrap.cc:492] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: No bootstrap required, opened a new log
I20260128 20:50:36.628770 7105 tablet_bootstrap.cc:492] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: No bootstrap required, opened a new log
I20260128 20:50:36.629040 7106 ts_tablet_manager.cc:1403] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260128 20:50:36.629040 7105 ts_tablet_manager.cc:1403] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260128 20:50:36.629040 7107 ts_tablet_manager.cc:1403] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:50:36.632642 7106 raft_consensus.cc:359] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.633108 7106 raft_consensus.cc:385] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:36.633230 7106 raft_consensus.cc:740] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36252231303540e2bad76a9e9696d0c1, State: Initialized, Role: FOLLOWER
I20260128 20:50:36.633009 7107 raft_consensus.cc:359] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.633009 7105 raft_consensus.cc:359] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.633383 7105 raft_consensus.cc:385] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:36.633394 7107 raft_consensus.cc:385] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:36.633476 7107 raft_consensus.cc:740] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Initialized, Role: FOLLOWER
I20260128 20:50:36.633476 7105 raft_consensus.cc:740] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Initialized, Role: FOLLOWER
I20260128 20:50:36.634084 7107 consensus_queue.cc:260] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.634085 7105 consensus_queue.cc:260] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.634559 7106 consensus_queue.cc:260] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.635222 7105 ts_tablet_manager.cc:1434] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260128 20:50:36.635214 7107 ts_tablet_manager.cc:1434] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260128 20:50:36.635694 7082 heartbeater.cc:499] Master 127.4.183.190:33587 was elected leader, sending a full tablet report...
I20260128 20:50:36.635702 6798 heartbeater.cc:499] Master 127.4.183.190:33587 was elected leader, sending a full tablet report...
I20260128 20:50:36.636303 6940 heartbeater.cc:499] Master 127.4.183.190:33587 was elected leader, sending a full tablet report...
I20260128 20:50:36.636723 7106 ts_tablet_manager.cc:1434] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260128 20:50:36.724447 7111 raft_consensus.cc:493] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:36.724673 7111 raft_consensus.cc:515] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
W20260128 20:50:36.725430 6799 tablet.cc:2378] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:50:36.726271 7111 leader_election.cc:290] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683), 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:36.730499 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.726642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:46683 (local address 127.4.183.129:39989)
0128 20:50:36.727436 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.727452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.727593 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.727857 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.727859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.727881 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:36.728165 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.728172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.729093 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.729096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.729959 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.729970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.730118 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.730140 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.730197 (+ 57us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.730284 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":59,"thread_start_us":147,"threads_started":1}
I20260128 20:50:36.730505 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.726642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:44061 (local address 127.4.183.129:32777)
0128 20:50:36.727139 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.727155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.727282 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.727674 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.727677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.727700 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:36.728032 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.728040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.729031 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.729037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.729962 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.729972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.730115 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.730141 (+ 26us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.730198 (+ 57us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.730284 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":158,"threads_started":1}
I20260128 20:50:36.730942 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.726753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:32777 (local address 127.4.183.131:44061)
0128 20:50:36.726983 (+ 230us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.726988 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.727241 (+ 253us) server_negotiation.cc:408] Connection header received
0128 20:50:36.727432 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.727436 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.727508 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.727613 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:36.728168 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.728860 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.730095 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.730651 (+ 556us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.730700 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.730759 (+ 59us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.730825 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260128 20:50:36.731047 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.726785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:39989 (local address 127.4.183.130:46683)
0128 20:50:36.726954 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.726959 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.727574 (+ 615us) server_negotiation.cc:408] Connection header received
0128 20:50:36.727682 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.727685 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.727740 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.727840 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:36.728326 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.728963 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.730095 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.730720 (+ 625us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.730766 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.730828 (+ 62us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.730908 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 20:50:36.731881 7036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b4a82f6a359489097cd65f38ab6e104" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506914e417bf4007985e767b4ded9e03" is_pre_election: true
I20260128 20:50:36.731880 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b4a82f6a359489097cd65f38ab6e104" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1" is_pre_election: true
I20260128 20:50:36.732218 6894 raft_consensus.cc:2468] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 0.
I20260128 20:50:36.732218 7036 raft_consensus.cc:2468] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 0.
I20260128 20:50:36.732905 6685 leader_election.cc:304] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36252231303540e2bad76a9e9696d0c1, 4b934f8638a849d99e8019d77f5d406d, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:36.733296 7111 raft_consensus.cc:2804] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:36.733400 7111 raft_consensus.cc:493] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:36.733453 7111 raft_consensus.cc:3060] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:36.734534 7111 raft_consensus.cc:515] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
I20260128 20:50:36.735024 7111 leader_election.cc:290] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 election: Requested vote from peers 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683), 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:36.735589 7036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b4a82f6a359489097cd65f38ab6e104" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506914e417bf4007985e767b4ded9e03"
I20260128 20:50:36.735785 7036 raft_consensus.cc:3060] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:36.735880 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b4a82f6a359489097cd65f38ab6e104" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1"
I20260128 20:50:36.736219 6894 raft_consensus.cc:3060] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:36.737455 6894 raft_consensus.cc:2468] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 1.
I20260128 20:50:36.737664 7036 raft_consensus.cc:2468] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 1.
I20260128 20:50:36.737963 6684 leader_election.cc:304] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36252231303540e2bad76a9e9696d0c1, 4b934f8638a849d99e8019d77f5d406d; no voters:
I20260128 20:50:36.738236 7111 raft_consensus.cc:2804] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:36.738533 7111 raft_consensus.cc:697] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 1 LEADER]: Becoming Leader. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Running, Role: LEADER
I20260128 20:50:36.738916 7111 consensus_queue.cc:237] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } }
W20260128 20:50:36.739918 6941 tablet.cc:2378] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:50:36.743314 6593 catalog_manager.cc:5654] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b934f8638a849d99e8019d77f5d406d (127.4.183.129). New cstate: current_term: 1 leader_uuid: "4b934f8638a849d99e8019d77f5d406d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } health_report { overall_health: HEALTHY } } }
W20260128 20:50:36.782917 7083 tablet.cc:2378] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 28 20:50:36 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633435, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:36.805418 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.795321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33587 (local address 127.0.0.1:36264)
0128 20:50:36.795663 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:36.795681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:36.795806 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:36.796077 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:36.796079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:36.796563 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:36.796775 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.796783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.797731 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.797735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:36.798144 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:36.798152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.798267 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.798885 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:36.798900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:36.800821 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:36.803012 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.803024 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.803029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.803320 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.803642 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:36.803644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:36.803645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:36.803698 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:36.804494 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:36.804505 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:36.804661 (+ 156us) client_negotiation.cc:769] Sending connection context
0128 20:50:36.804804 (+ 143us) client_negotiation.cc:240] Negotiation successful
0128 20:50:36.805043 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":142,"threads_started":1}
I20260128 20:50:36.805694 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:36.795449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.4.183.190:33587)
0128 20:50:36.795609 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:36.795613 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:36.795768 (+ 155us) server_negotiation.cc:408] Connection header received
0128 20:50:36.795892 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:36.795895 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:36.795942 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:36.796030 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:36.796937 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.797580 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:36.798268 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:36.798402 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:36.801021 (+ 2619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:36.801044 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:36.801046 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:36.801075 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:36.802820 (+ 1745us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.803497 (+ 677us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.803501 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.803502 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.803547 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:36.803888 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:36.803893 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:36.803895 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:36.804248 (+ 353us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:36.804417 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:36.804921 (+ 504us) server_negotiation.cc:300] Negotiation successful
0128 20:50:36.805335 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":53}
I20260128 20:50:36.811853 6593 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:36264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260128 20:50:36.812883 6593 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:36.820144 6874 tablet_service.cc:1505] Processing CreateTablet for tablet 4fe736065fea4491850690d7e76fa69c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b83e92b0e9c42ad9da04d1dc1693c82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:50:36.820125 6732 tablet_service.cc:1505] Processing CreateTablet for tablet 4fe736065fea4491850690d7e76fa69c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b83e92b0e9c42ad9da04d1dc1693c82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:50:36.820417 7016 tablet_service.cc:1505] Processing CreateTablet for tablet 4fe736065fea4491850690d7e76fa69c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b83e92b0e9c42ad9da04d1dc1693c82]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:50:36.820931 6874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe736065fea4491850690d7e76fa69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:36.820925 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe736065fea4491850690d7e76fa69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:36.820925 6732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe736065fea4491850690d7e76fa69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:36.824012 7105 tablet_bootstrap.cc:492] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: Bootstrap starting.
I20260128 20:50:36.824012 7107 tablet_bootstrap.cc:492] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: Bootstrap starting.
I20260128 20:50:36.824012 7106 tablet_bootstrap.cc:492] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: Bootstrap starting.
I20260128 20:50:36.825035 7106 tablet_bootstrap.cc:654] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:36.825428 7105 tablet_bootstrap.cc:654] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:36.825448 7107 tablet_bootstrap.cc:654] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:36.827216 7107 tablet_bootstrap.cc:492] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: No bootstrap required, opened a new log
I20260128 20:50:36.827212 7106 tablet_bootstrap.cc:492] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: No bootstrap required, opened a new log
I20260128 20:50:36.827347 7105 tablet_bootstrap.cc:492] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: No bootstrap required, opened a new log
I20260128 20:50:36.827463 7105 ts_tablet_manager.cc:1403] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260128 20:50:36.827455 7106 ts_tablet_manager.cc:1403] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:50:36.827447 7107 ts_tablet_manager.cc:1403] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260128 20:50:36.827981 7105 raft_consensus.cc:359] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:36.828110 7105 raft_consensus.cc:385] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:36.828070 7106 raft_consensus.cc:359] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:36.828152 7105 raft_consensus.cc:740] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Initialized, Role: FOLLOWER
I20260128 20:50:36.828181 7106 raft_consensus.cc:385] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:36.828217 7106 raft_consensus.cc:740] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36252231303540e2bad76a9e9696d0c1, State: Initialized, Role: FOLLOWER
I20260128 20:50:36.828305 7107 raft_consensus.cc:359] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:36.828369 7106 consensus_queue.cc:260] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:36.828437 7107 raft_consensus.cc:385] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:36.828474 7107 raft_consensus.cc:740] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Initialized, Role: FOLLOWER
I20260128 20:50:36.828289 7105 consensus_queue.cc:260] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:36.828677 7106 ts_tablet_manager.cc:1434] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260128 20:50:36.828644 7107 consensus_queue.cc:260] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:36.828819 7105 ts_tablet_manager.cc:1434] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260128 20:50:36.829034 7112 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:36.829480 7112 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:36.829511 7111 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:36.829659 7111 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:36.829824 7107 ts_tablet_manager.cc:1434] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260128 20:50:36.829034 7113 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:36.830255 7113 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:37.137354 7111 consensus_queue.cc:1048] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Connected to new peer: Peer: permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:37.145185 7116 consensus_queue.cc:1048] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Connected to new peer: Peer: permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:37.273751 7112 raft_consensus.cc:493] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:37.274075 7112 raft_consensus.cc:515] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.275550 7112 leader_election.cc:290] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279), 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683)
I20260128 20:50:37.280826 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.276063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:46683 (local address 127.4.183.131:35215)
0128 20:50:37.276923 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:37.276941 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:37.277167 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:37.277579 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:37.277582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:37.277607 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:37.277943 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.277953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.279119 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.279124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:37.280239 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.280255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.280426 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.280456 (+ 30us) client_negotiation.cc:769] Sending connection context
0128 20:50:37.280532 (+ 76us) client_negotiation.cc:240] Negotiation successful
0128 20:50:37.280605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":24,"thread_start_us":132,"threads_started":1}
I20260128 20:50:37.280826 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.276063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:44279 (local address 127.4.183.131:46717)
0128 20:50:37.276564 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:37.276580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:37.276734 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:37.277223 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:37.277229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:37.277260 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:50:37.277651 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.277661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.279119 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.279124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:37.280240 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.280255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.280426 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.280457 (+ 31us) client_negotiation.cc:769] Sending connection context
0128 20:50:37.280516 (+ 59us) client_negotiation.cc:240] Negotiation successful
0128 20:50:37.280605 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"spinlock_wait_cycles":26880,"thread_start_us":143,"threads_started":1}
I20260128 20:50:37.281487 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.276472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:35215 (local address 127.4.183.130:46683)
0128 20:50:37.276746 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:37.276752 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:37.277187 (+ 435us) server_negotiation.cc:408] Connection header received
0128 20:50:37.277306 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:37.277309 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:37.277390 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:37.277525 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:37.278172 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.278924 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.280507 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.281083 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.281161 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:37.281237 (+ 76us) server_negotiation.cc:300] Negotiation successful
0128 20:50:37.281320 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260128 20:50:37.281476 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.276238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:46717 (local address 127.4.183.129:44279)
0128 20:50:37.276737 (+ 499us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:37.276744 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:37.276771 (+ 27us) server_negotiation.cc:408] Connection header received
0128 20:50:37.276864 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:37.276869 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:37.276949 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:37.277116 (+ 167us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:50:37.277846 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.278834 (+ 988us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.280505 (+ 1671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.281083 (+ 578us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.281162 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:37.281264 (+ 102us) server_negotiation.cc:300] Negotiation successful
0128 20:50:37.281349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":226,"threads_started":1}
I20260128 20:50:37.281972 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fe736065fea4491850690d7e76fa69c" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1" is_pre_election: true
I20260128 20:50:37.282150 6894 raft_consensus.cc:2468] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 0.
I20260128 20:50:37.282150 6752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fe736065fea4491850690d7e76fa69c" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b934f8638a849d99e8019d77f5d406d" is_pre_election: true
I20260128 20:50:37.282466 6752 raft_consensus.cc:2468] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 0.
I20260128 20:50:37.282634 6968 leader_election.cc:304] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36252231303540e2bad76a9e9696d0c1, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:37.282950 7112 raft_consensus.cc:2804] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:37.283027 7112 raft_consensus.cc:493] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:37.283057 7112 raft_consensus.cc:3060] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:37.354594 7112 raft_consensus.cc:515] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.355324 7112 leader_election.cc:290] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 election: Requested vote from peers 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279), 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683)
I20260128 20:50:37.356252 6752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fe736065fea4491850690d7e76fa69c" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b934f8638a849d99e8019d77f5d406d"
I20260128 20:50:37.356258 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4fe736065fea4491850690d7e76fa69c" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1"
I20260128 20:50:37.356480 6752 raft_consensus.cc:3060] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:37.356509 6894 raft_consensus.cc:3060] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:37.357877 6894 raft_consensus.cc:2468] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 1.
I20260128 20:50:37.358355 6752 raft_consensus.cc:2468] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 1.
I20260128 20:50:37.358570 6968 leader_election.cc:304] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36252231303540e2bad76a9e9696d0c1, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:37.358908 7112 raft_consensus.cc:2804] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:37.359150 7112 raft_consensus.cc:697] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [term 1 LEADER]: Becoming Leader. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Running, Role: LEADER
I20260128 20:50:37.359550 7112 consensus_queue.cc:237] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.361047 7137 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: TxnStatusTablet state changed. Reason: New leader 506914e417bf4007985e767b4ded9e03. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.361279 7137 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:50:37.361603 7139 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260128 20:50:37.361769 7139 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260128 20:50:37.362640 6591 catalog_manager.cc:5654] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 506914e417bf4007985e767b4ded9e03 (127.4.183.131). New cstate: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } health_report { overall_health: HEALTHY } } }
I20260128 20:50:37.404381 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.399313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.4.183.131:44061)
0128 20:50:37.399662 (+ 349us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:37.399665 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:37.399689 (+ 24us) server_negotiation.cc:408] Connection header received
0128 20:50:37.399978 (+ 289us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:37.399981 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:37.400044 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:37.400248 (+ 204us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:37.400738 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.401471 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.402566 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.402795 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.402898 (+ 103us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.403806 (+ 908us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.403932 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:37.404126 (+ 194us) server_negotiation.cc:300] Negotiation successful
0128 20:50:37.404219 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260128 20:50:37.404750 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.399157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:44061 (local address 127.0.0.1:49648)
0128 20:50:37.399603 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:37.399622 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:37.399799 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:37.400257 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:37.400261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:37.400272 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:37.400587 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.400598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.401610 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.401615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:37.402331 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.402343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.402491 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.402561 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.403957 (+ 1396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.403981 (+ 24us) client_negotiation.cc:769] Sending connection context
0128 20:50:37.404075 (+ 94us) client_negotiation.cc:240] Negotiation successful
0128 20:50:37.404493 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":157,"threads_started":1}
I20260128 20:50:37.410691 6894 raft_consensus.cc:1275] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Refusing update from remote peer 506914e417bf4007985e767b4ded9e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:37.410838 6752 raft_consensus.cc:1275] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Refusing update from remote peer 506914e417bf4007985e767b4ded9e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:37.411338 7137 consensus_queue.cc:1048] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:37.411527 7112 consensus_queue.cc:1048] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:37.415071 7116 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: TxnStatusTablet state changed. Reason: New leader 506914e417bf4007985e767b4ded9e03. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.415251 7116 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:37.419584 7113 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: TxnStatusTablet state changed. Reason: New leader 506914e417bf4007985e767b4ded9e03. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.419764 7113 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:37.420850 7112 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.420993 7112 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:50:37.421540 7116 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.421669 7116 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 4b934f8638a849d99e8019d77f5d406d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:37.421765 7128 mvcc.cc:204] Tried to move back new op lower bound from 7248418559626129408 to 7248418559428169728. Current Snapshot: MvccSnapshot[applied={T|T < 7248418559626129408}]
I20260128 20:50:37.422905 7113 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.423024 7113 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 36252231303540e2bad76a9e9696d0c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:50:37.421000 7137 tablet_replica.cc:442] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } } }
I20260128 20:50:37.425112 7137 tablet_replica.cc:445] T 4fe736065fea4491850690d7e76fa69c P 506914e417bf4007985e767b4ded9e03: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:50:37.426765 7132 mvcc.cc:204] Tried to move back new op lower bound from 7248418559626129408 to 7248418559428169728. Current Snapshot: MvccSnapshot[applied={T|T < 7248418559626129408 or (T in {7248418559626129408})}]
I20260128 20:50:37.436874 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.430473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.4.183.129:44279)
0128 20:50:37.430633 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:37.430636 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:37.431315 (+ 679us) server_negotiation.cc:408] Connection header received
0128 20:50:37.432184 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:37.432189 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:37.432299 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:37.432447 (+ 148us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:37.433069 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.433708 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.435364 (+ 1656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.435645 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.435725 (+ 80us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.436315 (+ 590us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.436506 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:37.436607 (+ 101us) server_negotiation.cc:300] Negotiation successful
0128 20:50:37.436688 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260128 20:50:37.452566 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.446997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:44061 (local address 127.4.183.129:59169)
0128 20:50:37.447414 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:37.447431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:37.447555 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:37.447887 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:37.447890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:37.447906 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:37.448143 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.448149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.450592 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.450598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:37.451337 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.451360 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.451548 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.451608 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.452203 (+ 595us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.452212 (+ 9us) client_negotiation.cc:769] Sending connection context
0128 20:50:37.452298 (+ 86us) client_negotiation.cc:240] Negotiation successful
0128 20:50:37.452403 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260128 20:50:37.452889 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.447268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:59169 (local address 127.4.183.131:44061)
0128 20:50:37.447433 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:37.447438 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:37.447507 (+ 69us) server_negotiation.cc:408] Connection header received
0128 20:50:37.447666 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:37.447670 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:37.447735 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:37.447826 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:37.448293 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.449084 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.451575 (+ 2491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.451823 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.451885 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.451990 (+ 105us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.452096 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:37.452335 (+ 239us) server_negotiation.cc:300] Negotiation successful
0128 20:50:37.452569 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260128 20:50:37.487814 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.483659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:48787 (local address 127.4.183.129:44279)
0128 20:50:37.483825 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:37.483830 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:37.484098 (+ 268us) server_negotiation.cc:408] Connection header received
0128 20:50:37.484245 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:37.484249 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:37.484321 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:37.484428 (+ 107us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:37.485390 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.486048 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.486957 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.487168 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.487261 (+ 93us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.487363 (+ 102us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.487449 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:37.487581 (+ 132us) server_negotiation.cc:300] Negotiation successful
0128 20:50:37.487637 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260128 20:50:37.487833 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:37.483536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:44279 (local address 127.4.183.131:48787)
0128 20:50:37.483998 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:37.484013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:37.484132 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:37.484533 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:37.484536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:37.484545 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:37.485212 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.485226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.486206 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:37.486209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:37.486826 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:37.486836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:37.486975 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:37.487033 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:37.487480 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:37.487490 (+ 10us) client_negotiation.cc:769] Sending connection context
0128 20:50:37.487569 (+ 79us) client_negotiation.cc:240] Negotiation successful
0128 20:50:37.487662 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":120,"threads_started":1}
I20260128 20:50:37.526392 6593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:37.526612 6593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:37.529795 6593 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Sending DeleteTablet for 3 replicas of tablet 2b4a82f6a359489097cd65f38ab6e104
I20260128 20:50:37.530979 7016 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b4a82f6a359489097cd65f38ab6e104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:37 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:49638
I20260128 20:50:37.531877 6732 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b4a82f6a359489097cd65f38ab6e104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:37 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:43378
I20260128 20:50:37.532984 6874 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b4a82f6a359489097cd65f38ab6e104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:37 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:53562
I20260128 20:50:37.534796 6593 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:50:37.535360 6593 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:37.536711 7164 tablet_replica.cc:333] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: stopping tablet replica
I20260128 20:50:37.537169 7164 raft_consensus.cc:2243] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:50:37.537689 7164 raft_consensus.cc:2272] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:37.541642 7164 ts_tablet_manager.cc:1916] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:37.544039 6732 tablet_service.cc:1505] Processing CreateTablet for tablet b265d0937dbe4e358d67895539035565 (DEFAULT_TABLE table=test-table [id=3907a0eb4753481b8c7c4ed755c23c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:37.544430 6732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265d0937dbe4e358d67895539035565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:37.547857 7016 tablet_service.cc:1505] Processing CreateTablet for tablet b265d0937dbe4e358d67895539035565 (DEFAULT_TABLE table=test-table [id=3907a0eb4753481b8c7c4ed755c23c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:37.548214 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265d0937dbe4e358d67895539035565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:37.551983 6874 tablet_service.cc:1505] Processing CreateTablet for tablet b265d0937dbe4e358d67895539035565 (DEFAULT_TABLE table=test-table [id=3907a0eb4753481b8c7c4ed755c23c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:37.552652 6874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265d0937dbe4e358d67895539035565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:37.553733 7163 tablet_replica.cc:333] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: stopping tablet replica
I20260128 20:50:37.554108 7163 raft_consensus.cc:2243] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:37.554339 7163 raft_consensus.cc:2272] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:37.554714 7165 tablet_replica.cc:333] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: stopping tablet replica
I20260128 20:50:37.554980 7165 raft_consensus.cc:2243] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:37.555208 7165 raft_consensus.cc:2272] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:37.558791 7163 ts_tablet_manager.cc:1916] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:37.561604 7167 tablet_bootstrap.cc:492] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Bootstrap starting.
I20260128 20:50:37.563210 7167 tablet_bootstrap.cc:654] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:37.557102 7165 ts_tablet_manager.cc:1916] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:37.570803 7166 tablet_bootstrap.cc:492] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Bootstrap starting.
I20260128 20:50:37.573760 7166 tablet_bootstrap.cc:654] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:37.575418 7169 tablet_bootstrap.cc:492] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Bootstrap starting.
I20260128 20:50:37.575678 7167 tablet_bootstrap.cc:492] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: No bootstrap required, opened a new log
I20260128 20:50:37.575951 7167 ts_tablet_manager.cc:1403] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260128 20:50:37.576619 7167 raft_consensus.cc:359] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.576898 7167 raft_consensus.cc:385] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:37.576977 7167 raft_consensus.cc:740] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Initialized, Role: FOLLOWER
I20260128 20:50:37.577279 7167 consensus_queue.cc:260] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.577754 7167 ts_tablet_manager.cc:1434] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260128 20:50:37.577481 7169 tablet_bootstrap.cc:654] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:37.582252 7169 tablet_bootstrap.cc:492] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: No bootstrap required, opened a new log
I20260128 20:50:37.582396 7169 ts_tablet_manager.cc:1403] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260128 20:50:37.583621 7169 raft_consensus.cc:359] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.583774 7169 raft_consensus.cc:385] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:37.583811 7169 raft_consensus.cc:740] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36252231303540e2bad76a9e9696d0c1, State: Initialized, Role: FOLLOWER
I20260128 20:50:37.583983 7169 consensus_queue.cc:260] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.584578 7169 ts_tablet_manager.cc:1434] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260128 20:50:37.588111 7166 tablet_bootstrap.cc:492] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: No bootstrap required, opened a new log
I20260128 20:50:37.588326 7166 ts_tablet_manager.cc:1403] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20260128 20:50:37.589279 7166 raft_consensus.cc:359] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.589504 7166 raft_consensus.cc:385] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:37.589541 7166 raft_consensus.cc:740] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Initialized, Role: FOLLOWER
I20260128 20:50:37.589735 7166 consensus_queue.cc:260] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.590111 7166 ts_tablet_manager.cc:1434] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260128 20:50:37.659348 7129 raft_consensus.cc:493] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:37.659534 7129 raft_consensus.cc:515] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.660187 7129 leader_election.cc:290] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683), 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:37.661823 7036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b265d0937dbe4e358d67895539035565" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506914e417bf4007985e767b4ded9e03" is_pre_election: true
I20260128 20:50:37.661989 7036 raft_consensus.cc:2468] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 0.
I20260128 20:50:37.662436 6685 leader_election.cc:304] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b934f8638a849d99e8019d77f5d406d, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:37.663095 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b265d0937dbe4e358d67895539035565" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1" is_pre_election: true
I20260128 20:50:37.663149 7129 raft_consensus.cc:2804] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:37.663229 7129 raft_consensus.cc:493] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:37.663257 6894 raft_consensus.cc:2468] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 0.
I20260128 20:50:37.663280 7129 raft_consensus.cc:3060] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:37.664674 7129 raft_consensus.cc:515] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.665160 7129 leader_election.cc:290] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 election: Requested vote from peers 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683), 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:37.665827 7036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b265d0937dbe4e358d67895539035565" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506914e417bf4007985e767b4ded9e03"
I20260128 20:50:37.665966 7036 raft_consensus.cc:3060] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:37.666085 7165 ts_tablet_manager.cc:1929] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:37.666121 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b265d0937dbe4e358d67895539035565" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1"
I20260128 20:50:37.666241 7165 log.cc:1199] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/2b4a82f6a359489097cd65f38ab6e104
I20260128 20:50:37.666255 6894 raft_consensus.cc:3060] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:37.666646 7165 ts_tablet_manager.cc:1950] T 2b4a82f6a359489097cd65f38ab6e104 P 36252231303540e2bad76a9e9696d0c1: Deleting consensus metadata
I20260128 20:50:37.681690 7164 ts_tablet_manager.cc:1929] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:37.681885 7164 log.cc:1199] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/2b4a82f6a359489097cd65f38ab6e104
I20260128 20:50:37.682260 7164 ts_tablet_manager.cc:1950] T 2b4a82f6a359489097cd65f38ab6e104 P 4b934f8638a849d99e8019d77f5d406d: Deleting consensus metadata
I20260128 20:50:37.693825 6578 catalog_manager.cc:4985] TS 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683): tablet 2b4a82f6a359489097cd65f38ab6e104 (table test-table [id=28fd3b10c0f7413ba1e3fb09df0aafaa]) successfully deleted
I20260128 20:50:37.694103 6579 catalog_manager.cc:4985] TS 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279): tablet 2b4a82f6a359489097cd65f38ab6e104 (table test-table [id=28fd3b10c0f7413ba1e3fb09df0aafaa]) successfully deleted
I20260128 20:50:37.755352 7036 raft_consensus.cc:2468] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 1.
I20260128 20:50:37.755470 6894 raft_consensus.cc:2468] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 1.
I20260128 20:50:37.756202 7163 ts_tablet_manager.cc:1929] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:37.756353 7163 log.cc:1199] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/2b4a82f6a359489097cd65f38ab6e104
I20260128 20:50:37.756269 6685 leader_election.cc:304] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b934f8638a849d99e8019d77f5d406d, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:37.756629 7129 raft_consensus.cc:2804] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:37.756734 7129 raft_consensus.cc:697] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 1 LEADER]: Becoming Leader. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Running, Role: LEADER
I20260128 20:50:37.756791 7163 ts_tablet_manager.cc:1950] T 2b4a82f6a359489097cd65f38ab6e104 P 506914e417bf4007985e767b4ded9e03: Deleting consensus metadata
I20260128 20:50:37.756950 7129 consensus_queue.cc:237] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:37.759938 6591 catalog_manager.cc:5654] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b934f8638a849d99e8019d77f5d406d (127.4.183.129). New cstate: current_term: 1 leader_uuid: "4b934f8638a849d99e8019d77f5d406d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20260128 20:50:37.762470 6579 catalog_manager.cc:4985] TS 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061): tablet 2b4a82f6a359489097cd65f38ab6e104 (table test-table [id=28fd3b10c0f7413ba1e3fb09df0aafaa]) successfully deleted
I20260128 20:50:37.833922 7036 raft_consensus.cc:1275] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Refusing update from remote peer 4b934f8638a849d99e8019d77f5d406d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:37.834455 6894 raft_consensus.cc:1275] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Refusing update from remote peer 4b934f8638a849d99e8019d77f5d406d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:37.834789 7158 consensus_queue.cc:1048] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Connected to new peer: Peer: permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:37.835482 7129 consensus_queue.cc:1048] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Connected to new peer: Peer: permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260128 20:50:37.852909 7128 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260128 20:50:39.433277 6593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:39.433467 6593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:39.437247 6593 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Sending DeleteTablet for 3 replicas of tablet b265d0937dbe4e358d67895539035565
I20260128 20:50:39.438141 7016 tablet_service.cc:1552] Processing DeleteTablet for tablet b265d0937dbe4e358d67895539035565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:39 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:49638
I20260128 20:50:39.438141 6874 tablet_service.cc:1552] Processing DeleteTablet for tablet b265d0937dbe4e358d67895539035565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:39 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:53562
I20260128 20:50:39.438220 6732 tablet_service.cc:1552] Processing DeleteTablet for tablet b265d0937dbe4e358d67895539035565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:39 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:43378
I20260128 20:50:39.438578 7199 tablet_replica.cc:333] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: stopping tablet replica
I20260128 20:50:39.438585 7197 tablet_replica.cc:333] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: stopping tablet replica
I20260128 20:50:39.438683 7198 tablet_replica.cc:333] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: stopping tablet replica
I20260128 20:50:39.438722 7197 raft_consensus.cc:2243] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:39.438823 7198 raft_consensus.cc:2243] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:50:39.438889 7197 raft_consensus.cc:2272] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:39.439085 7199 raft_consensus.cc:2243] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:39.439121 7198 raft_consensus.cc:2272] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:39.439355 7199 raft_consensus.cc:2272] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:39.439751 7197 ts_tablet_manager.cc:1916] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:39.440253 7199 ts_tablet_manager.cc:1916] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:39.442255 7198 ts_tablet_manager.cc:1916] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:39.442853 7197 ts_tablet_manager.cc:1929] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 20:50:39.443013 7197 log.cc:1199] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/b265d0937dbe4e358d67895539035565
I20260128 20:50:39.443279 7199 ts_tablet_manager.cc:1929] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 20:50:39.443418 7199 log.cc:1199] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/b265d0937dbe4e358d67895539035565
I20260128 20:50:39.443415 7197 ts_tablet_manager.cc:1950] T b265d0937dbe4e358d67895539035565 P 506914e417bf4007985e767b4ded9e03: Deleting consensus metadata
I20260128 20:50:39.443797 7199 ts_tablet_manager.cc:1950] T b265d0937dbe4e358d67895539035565 P 36252231303540e2bad76a9e9696d0c1: Deleting consensus metadata
I20260128 20:50:39.444387 6579 catalog_manager.cc:4985] TS 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061): tablet b265d0937dbe4e358d67895539035565 (table test-table [id=3907a0eb4753481b8c7c4ed755c23c20]) successfully deleted
I20260128 20:50:39.445240 6578 catalog_manager.cc:4985] TS 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683): tablet b265d0937dbe4e358d67895539035565 (table test-table [id=3907a0eb4753481b8c7c4ed755c23c20]) successfully deleted
I20260128 20:50:39.445421 7198 ts_tablet_manager.cc:1929] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 20:50:39.445552 7198 log.cc:1199] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/b265d0937dbe4e358d67895539035565
I20260128 20:50:39.445890 7198 ts_tablet_manager.cc:1950] T b265d0937dbe4e358d67895539035565 P 4b934f8638a849d99e8019d77f5d406d: Deleting consensus metadata
I20260128 20:50:39.446733 6579 catalog_manager.cc:4985] TS 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279): tablet b265d0937dbe4e358d67895539035565 (table test-table [id=3907a0eb4753481b8c7c4ed755c23c20]) successfully deleted
Jan 28 20:50:39 dist-test-slave-hss7 krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 20:50:39 dist-test-slave-hss7 krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:39.476317 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:39.459912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.4.183.190:33587)
0128 20:50:39.460232 (+ 320us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:39.460235 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:39.460251 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:50:39.460298 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:39.460300 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:39.460350 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:39.460451 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:39.462006 (+ 1555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.462674 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:39.463572 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.463739 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:39.470317 (+ 6578us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:39.470346 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:39.470349 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:39.470395 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:39.472930 (+ 2535us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:39.473707 (+ 777us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:39.473712 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:39.473714 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:39.473793 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:39.474213 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:39.474217 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:39.474219 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:39.474605 (+ 386us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:39.474833 (+ 228us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:39.475603 (+ 770us) server_negotiation.cc:300] Negotiation successful
0128 20:50:39.475996 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
I20260128 20:50:39.481305 6593 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:50:39.481848 6593 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:39.487860 6732 tablet_service.cc:1505] Processing CreateTablet for tablet 53d9559d9467437988d31f5911d65a03 (DEFAULT_TABLE table=test-table [id=d41d128f3e924b7fb01a6a73baa70ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:39.488140 6874 tablet_service.cc:1505] Processing CreateTablet for tablet 53d9559d9467437988d31f5911d65a03 (DEFAULT_TABLE table=test-table [id=d41d128f3e924b7fb01a6a73baa70ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:39.488186 7016 tablet_service.cc:1505] Processing CreateTablet for tablet 53d9559d9467437988d31f5911d65a03 (DEFAULT_TABLE table=test-table [id=d41d128f3e924b7fb01a6a73baa70ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:39.488322 6732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d9559d9467437988d31f5911d65a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:39.488454 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d9559d9467437988d31f5911d65a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:39.488467 6874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d9559d9467437988d31f5911d65a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:39.490669 7208 tablet_bootstrap.cc:492] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Bootstrap starting.
I20260128 20:50:39.491250 7209 tablet_bootstrap.cc:492] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Bootstrap starting.
I20260128 20:50:39.491360 7210 tablet_bootstrap.cc:492] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Bootstrap starting.
I20260128 20:50:39.492303 7209 tablet_bootstrap.cc:654] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:39.492332 7208 tablet_bootstrap.cc:654] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:39.492432 7210 tablet_bootstrap.cc:654] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:39.493783 7209 tablet_bootstrap.cc:492] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: No bootstrap required, opened a new log
I20260128 20:50:39.493882 7208 tablet_bootstrap.cc:492] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: No bootstrap required, opened a new log
I20260128 20:50:39.493881 7210 tablet_bootstrap.cc:492] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: No bootstrap required, opened a new log
I20260128 20:50:39.493927 7209 ts_tablet_manager.cc:1403] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:50:39.493992 7210 ts_tablet_manager.cc:1403] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:50:39.494029 7208 ts_tablet_manager.cc:1403] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:39.494374 7209 raft_consensus.cc:359] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.494479 7209 raft_consensus.cc:385] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:39.494503 7209 raft_consensus.cc:740] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Initialized, Role: FOLLOWER
I20260128 20:50:39.494484 7210 raft_consensus.cc:359] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.494490 7208 raft_consensus.cc:359] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.494596 7210 raft_consensus.cc:385] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:39.494596 7208 raft_consensus.cc:385] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:39.494632 7210 raft_consensus.cc:740] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Initialized, Role: FOLLOWER
I20260128 20:50:39.494632 7208 raft_consensus.cc:740] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36252231303540e2bad76a9e9696d0c1, State: Initialized, Role: FOLLOWER
I20260128 20:50:39.494671 7209 consensus_queue.cc:260] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.494772 7210 consensus_queue.cc:260] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.494771 7208 consensus_queue.cc:260] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.495024 7209 ts_tablet_manager.cc:1434] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 20:50:39.495059 7210 ts_tablet_manager.cc:1434] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 20:50:39.495172 7208 ts_tablet_manager.cc:1434] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 20:50:39.613636 7173 raft_consensus.cc:493] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:39.613793 7173 raft_consensus.cc:515] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.614341 7173 leader_election.cc:290] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279), 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683)
I20260128 20:50:39.615118 6752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53d9559d9467437988d31f5911d65a03" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b934f8638a849d99e8019d77f5d406d" is_pre_election: true
I20260128 20:50:39.615146 6893 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53d9559d9467437988d31f5911d65a03" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1" is_pre_election: true
I20260128 20:50:39.615509 6752 raft_consensus.cc:2468] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 0.
I20260128 20:50:39.615533 6893 raft_consensus.cc:2468] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 0.
I20260128 20:50:39.616017 6968 leader_election.cc:304] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36252231303540e2bad76a9e9696d0c1, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:39.616255 7173 raft_consensus.cc:2804] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:39.616317 7173 raft_consensus.cc:493] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:39.616346 7173 raft_consensus.cc:3060] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:39.617583 7173 raft_consensus.cc:515] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.618063 7173 leader_election.cc:290] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 election: Requested vote from peers 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279), 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683)
I20260128 20:50:39.618564 6752 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53d9559d9467437988d31f5911d65a03" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b934f8638a849d99e8019d77f5d406d"
I20260128 20:50:39.618744 6752 raft_consensus.cc:3060] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:39.618673 6893 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53d9559d9467437988d31f5911d65a03" candidate_uuid: "506914e417bf4007985e767b4ded9e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1"
I20260128 20:50:39.618870 6893 raft_consensus.cc:3060] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:39.619813 6752 raft_consensus.cc:2468] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 1.
I20260128 20:50:39.619813 6893 raft_consensus.cc:2468] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506914e417bf4007985e767b4ded9e03 in term 1.
I20260128 20:50:39.620278 6969 leader_election.cc:304] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b934f8638a849d99e8019d77f5d406d, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:39.620507 7173 raft_consensus.cc:2804] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:39.620611 7173 raft_consensus.cc:697] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 1 LEADER]: Becoming Leader. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Running, Role: LEADER
I20260128 20:50:39.620764 7173 consensus_queue.cc:237] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.622793 6592 catalog_manager.cc:5654] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 506914e417bf4007985e767b4ded9e03 (127.4.183.131). New cstate: current_term: 1 leader_uuid: "506914e417bf4007985e767b4ded9e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } health_report { overall_health: HEALTHY } } }
I20260128 20:50:39.669826 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:39.666869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.4.183.131:44061)
0128 20:50:39.667201 (+ 332us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:39.667205 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:39.667226 (+ 21us) server_negotiation.cc:408] Connection header received
0128 20:50:39.667277 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:39.667280 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:39.667328 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:39.667436 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:39.667821 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.668338 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:39.669017 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.669235 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:39.669281 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:39.669363 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:39.669457 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:39.669593 (+ 136us) server_negotiation.cc:300] Negotiation successful
0128 20:50:39.669650 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
I20260128 20:50:39.681725 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:39.678203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:53289 (local address 127.4.183.131:44061)
0128 20:50:39.678333 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:39.678336 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:39.678515 (+ 179us) server_negotiation.cc:408] Connection header received
0128 20:50:39.678631 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:39.678633 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:39.678683 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:39.678783 (+ 100us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:39.679244 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.679983 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:39.680895 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.681077 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:39.681118 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:39.681264 (+ 146us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:39.681339 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:39.681544 (+ 205us) server_negotiation.cc:300] Negotiation successful
0128 20:50:39.681602 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 20:50:39.681751 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:39.678089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:44061 (local address 127.4.183.131:53289)
0128 20:50:39.678414 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:39.678444 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:39.678549 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:39.678800 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:39.678802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:39.678810 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:50:39.679016 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:39.679022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.680246 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:39.680250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:39.680735 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:39.680745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.680869 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:39.680897 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:50:39.681395 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:50:39.681401 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 20:50:39.681497 (+ 96us) client_negotiation.cc:240] Negotiation successful
0128 20:50:39.681580 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":141,"threads_started":1}
I20260128 20:50:39.688699 6752 raft_consensus.cc:1275] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Refusing update from remote peer 506914e417bf4007985e767b4ded9e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:39.688753 6893 raft_consensus.cc:1275] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Refusing update from remote peer 506914e417bf4007985e767b4ded9e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:39.689281 7183 consensus_queue.cc:1048] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:39.689522 7173 consensus_queue.cc:1048] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:39.731176 6590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:39.731384 6590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:39.734098 6590 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Sending DeleteTablet for 3 replicas of tablet 53d9559d9467437988d31f5911d65a03
I20260128 20:50:39.735256 6874 tablet_service.cc:1552] Processing DeleteTablet for tablet 53d9559d9467437988d31f5911d65a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:39 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:53562
I20260128 20:50:39.735445 7016 tablet_service.cc:1552] Processing DeleteTablet for tablet 53d9559d9467437988d31f5911d65a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:39 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:49638
I20260128 20:50:39.735453 6732 tablet_service.cc:1552] Processing DeleteTablet for tablet 53d9559d9467437988d31f5911d65a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:39 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:43378
I20260128 20:50:39.735658 7199 tablet_replica.cc:333] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: stopping tablet replica
I20260128 20:50:39.735697 7198 tablet_replica.cc:333] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: stopping tablet replica
I20260128 20:50:39.735700 7197 tablet_replica.cc:333] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: stopping tablet replica
I20260128 20:50:39.735842 7197 raft_consensus.cc:2243] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:50:39.735831 7198 raft_consensus.cc:2243] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:39.735831 7199 raft_consensus.cc:2243] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:39.735993 7198 raft_consensus.cc:2272] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:39.735993 7199 raft_consensus.cc:2272] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:39.736325 7197 raft_consensus.cc:2272] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:39.737251 7198 ts_tablet_manager.cc:1916] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:39.737596 7199 ts_tablet_manager.cc:1916] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:39.738219 7197 ts_tablet_manager.cc:1916] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:39.740289 7198 ts_tablet_manager.cc:1929] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:39.740280 7199 ts_tablet_manager.cc:1929] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:39.740382 7198 log.cc:1199] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/53d9559d9467437988d31f5911d65a03
I20260128 20:50:39.740382 7199 log.cc:1199] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/53d9559d9467437988d31f5911d65a03
I20260128 20:50:39.740686 7199 ts_tablet_manager.cc:1950] T 53d9559d9467437988d31f5911d65a03 P 36252231303540e2bad76a9e9696d0c1: Deleting consensus metadata
I20260128 20:50:39.740721 7198 ts_tablet_manager.cc:1950] T 53d9559d9467437988d31f5911d65a03 P 4b934f8638a849d99e8019d77f5d406d: Deleting consensus metadata
I20260128 20:50:39.740759 7197 ts_tablet_manager.cc:1929] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:50:39.740851 7197 log.cc:1199] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/53d9559d9467437988d31f5911d65a03
I20260128 20:50:39.741086 7197 ts_tablet_manager.cc:1950] T 53d9559d9467437988d31f5911d65a03 P 506914e417bf4007985e767b4ded9e03: Deleting consensus metadata
I20260128 20:50:39.741449 6590 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 20:50:39.741587 6579 catalog_manager.cc:4985] TS 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279): tablet 53d9559d9467437988d31f5911d65a03 (table test-table [id=d41d128f3e924b7fb01a6a73baa70ae9]) successfully deleted
I20260128 20:50:39.741882 6578 catalog_manager.cc:4985] TS 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683): tablet 53d9559d9467437988d31f5911d65a03 (table test-table [id=d41d128f3e924b7fb01a6a73baa70ae9]) successfully deleted
W20260128 20:50:39.742018 6590 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:50:39.743592 6579 catalog_manager.cc:4985] TS 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061): tablet 53d9559d9467437988d31f5911d65a03 (table test-table [id=d41d128f3e924b7fb01a6a73baa70ae9]) successfully deleted
I20260128 20:50:39.748276 6732 tablet_service.cc:1505] Processing CreateTablet for tablet deca1b3da5744863a89f7750041b95df (DEFAULT_TABLE table=test-table [id=86c0fdcf82db4326979a950b649ccfa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:39.748667 6732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deca1b3da5744863a89f7750041b95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:39.748670 6874 tablet_service.cc:1505] Processing CreateTablet for tablet deca1b3da5744863a89f7750041b95df (DEFAULT_TABLE table=test-table [id=86c0fdcf82db4326979a950b649ccfa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:39.748924 6874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deca1b3da5744863a89f7750041b95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:39.751516 7210 tablet_bootstrap.cc:492] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d: Bootstrap starting.
I20260128 20:50:39.751806 7016 tablet_service.cc:1505] Processing CreateTablet for tablet deca1b3da5744863a89f7750041b95df (DEFAULT_TABLE table=test-table [id=86c0fdcf82db4326979a950b649ccfa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:50:39.752105 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deca1b3da5744863a89f7750041b95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:39.752606 7210 tablet_bootstrap.cc:654] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:39.754040 7210 tablet_bootstrap.cc:492] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d: No bootstrap required, opened a new log
I20260128 20:50:39.754132 7210 ts_tablet_manager.cc:1403] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:50:39.754710 7210 raft_consensus.cc:359] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.754848 7210 raft_consensus.cc:385] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:39.754884 7210 raft_consensus.cc:740] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Initialized, Role: FOLLOWER
I20260128 20:50:39.755000 7208 tablet_bootstrap.cc:492] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Bootstrap starting.
I20260128 20:50:39.755026 7210 consensus_queue.cc:260] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.755334 7210 ts_tablet_manager.cc:1434] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 20:50:39.756139 7208 tablet_bootstrap.cc:654] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:39.757958 7208 tablet_bootstrap.cc:492] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: No bootstrap required, opened a new log
I20260128 20:50:39.758096 7208 ts_tablet_manager.cc:1403] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:39.758615 7208 raft_consensus.cc:359] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.758711 7208 raft_consensus.cc:385] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:39.758733 7208 raft_consensus.cc:740] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36252231303540e2bad76a9e9696d0c1, State: Initialized, Role: FOLLOWER
I20260128 20:50:39.758880 7208 consensus_queue.cc:260] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.759317 7208 ts_tablet_manager.cc:1434] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 20:50:39.759828 7209 tablet_bootstrap.cc:492] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Bootstrap starting.
I20260128 20:50:39.760960 7209 tablet_bootstrap.cc:654] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:39.762224 7209 tablet_bootstrap.cc:492] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: No bootstrap required, opened a new log
I20260128 20:50:39.762357 7209 ts_tablet_manager.cc:1403] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:50:39.762912 7209 raft_consensus.cc:359] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.763057 7209 raft_consensus.cc:385] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:39.763093 7209 raft_consensus.cc:740] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506914e417bf4007985e767b4ded9e03, State: Initialized, Role: FOLLOWER
I20260128 20:50:39.763257 7209 consensus_queue.cc:260] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.763588 7209 ts_tablet_manager.cc:1434] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 20:50:39.806301 7191 raft_consensus.cc:493] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:50:39.806452 7191 raft_consensus.cc:515] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.807430 7191 leader_election.cc:290] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683), 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:39.808257 7036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "deca1b3da5744863a89f7750041b95df" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506914e417bf4007985e767b4ded9e03" is_pre_election: true
I20260128 20:50:39.808316 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "deca1b3da5744863a89f7750041b95df" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1" is_pre_election: true
I20260128 20:50:39.808526 7036 raft_consensus.cc:2468] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 0.
I20260128 20:50:39.808579 6894 raft_consensus.cc:2468] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 0.
I20260128 20:50:39.809217 6685 leader_election.cc:304] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b934f8638a849d99e8019d77f5d406d, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:39.809525 7191 raft_consensus.cc:2804] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:50:39.809607 7191 raft_consensus.cc:493] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:50:39.809662 7191 raft_consensus.cc:3060] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:39.810724 7191 raft_consensus.cc:515] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.811235 7191 leader_election.cc:290] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 election: Requested vote from peers 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683), 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061)
I20260128 20:50:39.811764 7036 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "deca1b3da5744863a89f7750041b95df" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506914e417bf4007985e767b4ded9e03"
I20260128 20:50:39.811942 7036 raft_consensus.cc:3060] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:39.813740 6894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "deca1b3da5744863a89f7750041b95df" candidate_uuid: "4b934f8638a849d99e8019d77f5d406d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36252231303540e2bad76a9e9696d0c1"
I20260128 20:50:39.813858 7036 raft_consensus.cc:2468] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 1.
I20260128 20:50:39.813933 6894 raft_consensus.cc:3060] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:39.814440 6685 leader_election.cc:304] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b934f8638a849d99e8019d77f5d406d, 506914e417bf4007985e767b4ded9e03; no voters:
I20260128 20:50:39.814695 7191 raft_consensus.cc:2804] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:39.814817 7191 raft_consensus.cc:697] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [term 1 LEADER]: Becoming Leader. State: Replica: 4b934f8638a849d99e8019d77f5d406d, State: Running, Role: LEADER
I20260128 20:50:39.814978 7191 consensus_queue.cc:237] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } }
I20260128 20:50:39.815155 6894 raft_consensus.cc:2468] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b934f8638a849d99e8019d77f5d406d in term 1.
I20260128 20:50:39.817173 6592 catalog_manager.cc:5654] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b934f8638a849d99e8019d77f5d406d (127.4.183.129). New cstate: current_term: 1 leader_uuid: "4b934f8638a849d99e8019d77f5d406d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b934f8638a849d99e8019d77f5d406d" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 44279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20260128 20:50:39.854000 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:39.849847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.4.183.129:44279)
0128 20:50:39.850158 (+ 311us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:39.850161 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:39.850174 (+ 13us) server_negotiation.cc:408] Connection header received
0128 20:50:39.850245 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:39.850248 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:39.850296 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:39.850376 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:50:39.851276 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.851825 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:39.853049 (+ 1224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:39.853309 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:39.853366 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:50:39.853454 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:50:39.853526 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:39.853712 (+ 186us) server_negotiation.cc:300] Negotiation successful
0128 20:50:39.853803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":155,"threads_started":1}
I20260128 20:50:39.864791 6894 raft_consensus.cc:1275] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Refusing update from remote peer 4b934f8638a849d99e8019d77f5d406d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:39.865293 7036 raft_consensus.cc:1275] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Refusing update from remote peer 4b934f8638a849d99e8019d77f5d406d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:50:39.865515 7191 consensus_queue.cc:1048] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Connected to new peer: Peer: permanent_uuid: "36252231303540e2bad76a9e9696d0c1" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:50:39.866043 7191 consensus_queue.cc:1048] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d [LEADER]: Connected to new peer: Peer: permanent_uuid: "506914e417bf4007985e767b4ded9e03" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260128 20:50:39.871052 7179 mvcc.cc:204] Tried to move back new op lower bound from 7248418569682509824 to 7248418569483862016. Current Snapshot: MvccSnapshot[applied={T|T < 7248418569682509824}]
W20260128 20:50:39.886127 7180 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260128 20:50:41.467635 6590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:41.467828 6590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:50:41.470775 6590 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0ad5af48abd2405a950984ce55300382: Sending DeleteTablet for 3 replicas of tablet deca1b3da5744863a89f7750041b95df
I20260128 20:50:41.471616 6874 tablet_service.cc:1552] Processing DeleteTablet for tablet deca1b3da5744863a89f7750041b95df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:41 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:53562
I20260128 20:50:41.471638 6732 tablet_service.cc:1552] Processing DeleteTablet for tablet deca1b3da5744863a89f7750041b95df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:41 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:43378
I20260128 20:50:41.471689 7016 tablet_service.cc:1552] Processing DeleteTablet for tablet deca1b3da5744863a89f7750041b95df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:50:41 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:49638
I20260128 20:50:41.472055 7255 tablet_replica.cc:333] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: stopping tablet replica
I20260128 20:50:41.472062 7257 tablet_replica.cc:333] T deca1b3da5744863a89f7750041b95df P 4b934f8638a849d99e8019d77f5d406d: stopping tablet replica
I20260128 20:50:41.472092 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6667
I20260128 20:50:41.472146 7256 tablet_replica.cc:333] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: stopping tablet replica
I20260128 20:50:41.473301 7256 raft_consensus.cc:2243] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:41.473472 7256 raft_consensus.cc:2272] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:41.474105 7255 raft_consensus.cc:2243] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:50:41.474256 7255 raft_consensus.cc:2272] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:50:41.474308 7256 ts_tablet_manager.cc:1916] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:41.475533 7255 ts_tablet_manager.cc:1916] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:50:41.476615 7256 ts_tablet_manager.cc:1929] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 20:50:41.476717 7256 log.cc:1199] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/deca1b3da5744863a89f7750041b95df
I20260128 20:50:41.477046 7256 ts_tablet_manager.cc:1950] T deca1b3da5744863a89f7750041b95df P 506914e417bf4007985e767b4ded9e03: Deleting consensus metadata
I20260128 20:50:41.477908 6579 catalog_manager.cc:4985] TS 506914e417bf4007985e767b4ded9e03 (127.4.183.131:44061): tablet deca1b3da5744863a89f7750041b95df (table test-table [id=86c0fdcf82db4326979a950b649ccfa0]) successfully deleted
I20260128 20:50:41.478132 7255 ts_tablet_manager.cc:1929] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 20:50:41.478217 7255 log.cc:1199] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/deca1b3da5744863a89f7750041b95df
I20260128 20:50:41.478485 7255 ts_tablet_manager.cc:1950] T deca1b3da5744863a89f7750041b95df P 36252231303540e2bad76a9e9696d0c1: Deleting consensus metadata
I20260128 20:50:41.479342 6578 catalog_manager.cc:4985] TS 36252231303540e2bad76a9e9696d0c1 (127.4.183.130:46683): tablet deca1b3da5744863a89f7750041b95df (table test-table [id=86c0fdcf82db4326979a950b649ccfa0]) successfully deleted
W20260128 20:50:41.480593 6579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.183.129:44279 (error 108)
I20260128 20:50:41.480827 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6809
I20260128 20:50:41.483014 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:41.482074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:44279 (local address 127.0.0.1:42408)
0128 20:50:41.482785 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:41.482868 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.129:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":91,"threads_started":1}
W20260128 20:50:41.483234 6579 catalog_manager.cc:4712] TS 4b934f8638a849d99e8019d77f5d406d (127.4.183.129:44279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet deca1b3da5744863a89f7750041b95df: Network error: Client connection negotiation failed: client connection to 127.4.183.129:44279: connect: Connection refused (error 111)
I20260128 20:50:41.487598 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6951
I20260128 20:50:41.494869 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 6561
2026-01-28T20:50:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8266 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:41 dist-test-slave-hss7 krb5kdc[7262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:41 dist-test-slave-hss7 krb5kdc[7262](info): set up 2 sockets
Jan 28 20:50:41 dist-test-slave-hss7 krb5kdc[7262](info): commencing operation
krb5kdc: starting...
W20260128 20:50:43.551064 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.030s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:43 dist-test-slave-hss7 krb5kdc[7262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:43.717908 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:46455
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:43.838378 7278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:43.838676 7278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:43.838738 7278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:43.842571 7278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:43.842664 7278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:43.842689 7278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:43.842710 7278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:43.842727 7278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:43.847666 7278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:46455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:43.849011 7278 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:43.850010 7278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:43.856611 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:43.856611 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:43.856822 7278 server_base.cc:1060] running on GCE node
W20260128 20:50:43.856666 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:43.857475 7278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:43.858570 7278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:43.859764 7278 hybrid_clock.cc:648] HybridClock initialized: now 1769633443859737 us; error 44 us; skew 500 ppm
Jan 28 20:50:43 dist-test-slave-hss7 krb5kdc[7262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633443, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:43.862849 7278 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:43.864149 7278 webserver.cc:492] Webserver started at http://127.4.183.190:43929/ using document root <none> and password file <none>
I20260128 20:50:43.865012 7278 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:43.865079 7278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:43.865394 7278 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:43.867311 7278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "2c2fdf449c3049029ad6112dfcde07ae"
format_stamp: "Formatted at 2026-01-28 20:50:43 on dist-test-slave-hss7"
server_key: "f4a149d104776a7ff3b1c1ab96e8ecd8"
server_key_iv: "e6a4bddf32be85a3d7ffd11aeafed327"
server_key_version: "encryptionkey@0"
I20260128 20:50:43.867868 7278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "2c2fdf449c3049029ad6112dfcde07ae"
format_stamp: "Formatted at 2026-01-28 20:50:43 on dist-test-slave-hss7"
server_key: "f4a149d104776a7ff3b1c1ab96e8ecd8"
server_key_iv: "e6a4bddf32be85a3d7ffd11aeafed327"
server_key_version: "encryptionkey@0"
I20260128 20:50:43.871873 7278 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:50:43.874671 7293 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:43.875892 7278 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:50:43.876055 7278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "2c2fdf449c3049029ad6112dfcde07ae"
format_stamp: "Formatted at 2026-01-28 20:50:43 on dist-test-slave-hss7"
server_key: "f4a149d104776a7ff3b1c1ab96e8ecd8"
server_key_iv: "e6a4bddf32be85a3d7ffd11aeafed327"
server_key_version: "encryptionkey@0"
I20260128 20:50:43.876179 7278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:43.891868 7278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:43.895114 7278 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:43.895337 7278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:43.903616 7278 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:46455
I20260128 20:50:43.903614 7345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:46455 every 8 connection(s)
I20260128 20:50:43.904949 7278 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:43.905309 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 7278
I20260128 20:50:43.905475 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:43.905755 4830 external_mini_cluster.cc:1468] Setting key de8b63fb2e5d4055d99beb81bcc2c6f2
I20260128 20:50:43.908794 7346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jan 28 20:50:43 dist-test-slave-hss7 krb5kdc[7262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:43.915280 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae: Bootstrap starting.
I20260128 20:50:43.917930 7346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:43.918720 7346 log.cc:826] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:43.919914 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:43.907197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.4.183.190:46455)
0128 20:50:43.907628 (+ 431us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:43.907638 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:43.907675 (+ 37us) server_negotiation.cc:408] Connection header received
0128 20:50:43.908373 (+ 698us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:43.908393 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:43.908724 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:43.909111 (+ 387us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:43.910307 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:43.911146 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:43.911906 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:43.912242 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:43.914564 (+ 2322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:43.914619 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:43.914641 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:43.914692 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:43.916793 (+ 2101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:43.917409 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:43.917414 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:43.917420 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:43.917516 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:43.917797 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:43.917800 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:43.917801 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:43.918257 (+ 456us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:43.918520 (+ 263us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:43.918786 (+ 266us) server_negotiation.cc:300] Negotiation successful
0128 20:50:43.919094 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":142,"threads_started":1}
I20260128 20:50:43.920930 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae: No bootstrap required, opened a new log
I20260128 20:50:43.923852 7346 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2fdf449c3049029ad6112dfcde07ae" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 46455 } }
I20260128 20:50:43.924106 7346 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:43.924144 7346 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2fdf449c3049029ad6112dfcde07ae, State: Initialized, Role: FOLLOWER
I20260128 20:50:43.924652 7346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2fdf449c3049029ad6112dfcde07ae" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 46455 } }
I20260128 20:50:43.924784 7346 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:43.924901 7346 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:43.925025 7346 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:43.926072 7346 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2fdf449c3049029ad6112dfcde07ae" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 46455 } }
I20260128 20:50:43.926451 7346 leader_election.cc:304] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c2fdf449c3049029ad6112dfcde07ae; no voters:
I20260128 20:50:43.926772 7346 leader_election.cc:290] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:43.926956 7351 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:43.927215 7351 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [term 1 LEADER]: Becoming Leader. State: Replica: 2c2fdf449c3049029ad6112dfcde07ae, State: Running, Role: LEADER
I20260128 20:50:43.927584 7351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2fdf449c3049029ad6112dfcde07ae" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 46455 } }
I20260128 20:50:43.928632 7346 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:43.929860 7352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c2fdf449c3049029ad6112dfcde07ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2fdf449c3049029ad6112dfcde07ae" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 46455 } } }
I20260128 20:50:43.930015 7352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:43.930531 7353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c2fdf449c3049029ad6112dfcde07ae. Latest consensus state: current_term: 1 leader_uuid: "2c2fdf449c3049029ad6112dfcde07ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2fdf449c3049029ad6112dfcde07ae" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 46455 } } }
I20260128 20:50:43.930689 7353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae [sys.catalog]: This master's current role is: LEADER
W20260128 20:50:43.934036 7367 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:50:43.934165 7367 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:50:43.934273 7359 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:43.935082 7359 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:43.941610 7359 catalog_manager.cc:1357] Generated new cluster ID: df010db244354884bd7b249773f0c438
I20260128 20:50:43.941725 7359 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:43.967113 7359 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:43.968719 7359 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:43.984979 7359 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2c2fdf449c3049029ad6112dfcde07ae: Generated new TSK 0
I20260128 20:50:43.986188 7359 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:44.053617 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:46455
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:44.177326 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:44.177623 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:44.177693 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:44.181710 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:44.181805 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:44.181936 7374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:44.186924 7374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:44.188264 7374 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:44.189335 7374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:44.197109 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:44.197108 7379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:44.197124 7380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:44.197847 7374 server_base.cc:1060] running on GCE node
I20260128 20:50:44.198344 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:44.199136 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:44.200358 7374 hybrid_clock.cc:648] HybridClock initialized: now 1769633444200309 us; error 74 us; skew 500 ppm
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:44.203944 7374 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:44.205430 7374 webserver.cc:492] Webserver started at http://127.4.183.129:40611/ using document root <none> and password file <none>
I20260128 20:50:44.206097 7374 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:44.206183 7374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:44.206430 7374 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:44.208513 7374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "bee3061b3e5b4c5ea8f39ea31dbf0046"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "40af4d19c5491fbfd7e613428f27e1ae"
server_key_iv: "135977442f06a2835010cb839b14449e"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.209199 7374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "bee3061b3e5b4c5ea8f39ea31dbf0046"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "40af4d19c5491fbfd7e613428f27e1ae"
server_key_iv: "135977442f06a2835010cb839b14449e"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.213649 7374 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:44.216745 7389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.218529 7374 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:44.218735 7374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "bee3061b3e5b4c5ea8f39ea31dbf0046"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "40af4d19c5491fbfd7e613428f27e1ae"
server_key_iv: "135977442f06a2835010cb839b14449e"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.218880 7374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:44.239315 7374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:44.243759 7374 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:44.244024 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:44.244680 7374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:44.245883 7374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:44.245965 7374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.246037 7374 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:44.246096 7374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.257932 7374 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:42309
I20260128 20:50:44.257946 7502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:42309 every 8 connection(s)
I20260128 20:50:44.259120 7374 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:50:44.261795 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 7374
I20260128 20:50:44.261962 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:44.262475 4830 external_mini_cluster.cc:1468] Setting key 6a856733ef633595fdcc3968a50dcb84
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:50:44.277467 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:44.261449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:43349 (local address 127.4.183.190:46455)
0128 20:50:44.261621 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:44.261625 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:44.263230 (+ 1605us) server_negotiation.cc:408] Connection header received
0128 20:50:44.264405 (+ 1175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:44.264411 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:44.264484 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:44.264591 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:44.267300 (+ 2709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.267833 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:44.269029 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.269433 (+ 404us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:44.272199 (+ 2766us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:44.272228 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:44.272230 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:44.272268 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:44.274146 (+ 1878us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:44.274975 (+ 829us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:44.274982 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:44.274987 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:44.275062 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:44.275762 (+ 700us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:44.275766 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:44.275787 (+ 21us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:44.276051 (+ 264us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:44.276190 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:44.277058 (+ 868us) server_negotiation.cc:300] Negotiation successful
0128 20:50:44.277268 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260128 20:50:44.278239 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:44.262298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:46455 (local address 127.4.183.129:43349)
0128 20:50:44.263056 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:44.263117 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:44.264107 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:44.264865 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:44.264876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:44.265493 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:44.266958 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:44.266992 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.268012 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:44.268020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:44.268731 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:44.268747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.269063 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:44.269885 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:44.269937 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:44.271951 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:44.274350 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:44.274358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:44.274381 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:44.274716 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:44.275291 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:44.275297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:44.275299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:44.275523 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:44.276333 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:44.276345 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:44.276703 (+ 358us) client_negotiation.cc:769] Sending connection context
0128 20:50:44.276972 (+ 269us) client_negotiation.cc:240] Negotiation successful
0128 20:50:44.277359 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":234,"threads_started":1}
I20260128 20:50:44.280452 7503 heartbeater.cc:344] Connected to a master server at 127.4.183.190:46455
I20260128 20:50:44.280797 7503 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:44.281651 7503 heartbeater.cc:507] Master 127.4.183.190:46455 requested a full tablet report, sending...
I20260128 20:50:44.283636 7310 ts_manager.cc:194] Registered new tserver with Master: bee3061b3e5b4c5ea8f39ea31dbf0046 (127.4.183.129:42309)
I20260128 20:50:44.284971 7310 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:43349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:44.325773 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:46455
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:44.449781 7510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:44.450075 7510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:44.450165 7510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:44.453827 7510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:44.453948 7510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:44.454068 7510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:44.459069 7510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:44.460359 7510 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:44.461416 7510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:44.468983 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:44.468983 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:44.469071 7510 server_base.cc:1060] running on GCE node
W20260128 20:50:44.469074 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:44.469754 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:44.470443 7510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:44.471714 7510 hybrid_clock.cc:648] HybridClock initialized: now 1769633444471671 us; error 83 us; skew 500 ppm
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:44.475085 7510 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:44.476483 7510 webserver.cc:492] Webserver started at http://127.4.183.130:43817/ using document root <none> and password file <none>
I20260128 20:50:44.477236 7510 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:44.477325 7510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:44.477557 7510 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:44.479533 7510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "440341c0331d4e77805544e7a2929199"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "dee2bef9812b3a0d8abb837cef1f0523"
server_key_iv: "3eed17a2aeba063d8d36d81309bbfde4"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.480106 7510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "440341c0331d4e77805544e7a2929199"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "dee2bef9812b3a0d8abb837cef1f0523"
server_key_iv: "3eed17a2aeba063d8d36d81309bbfde4"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.484748 7510 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260128 20:50:44.488600 7525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.490113 7510 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:44.490330 7510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "440341c0331d4e77805544e7a2929199"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "dee2bef9812b3a0d8abb837cef1f0523"
server_key_iv: "3eed17a2aeba063d8d36d81309bbfde4"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.490494 7510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:44.505962 7510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:44.513477 7510 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:44.513723 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:44.514351 7510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:44.515326 7510 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:44.515401 7510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.515475 7510 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:44.515532 7510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.526551 7510 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:39809
I20260128 20:50:44.526584 7638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:39809 every 8 connection(s)
I20260128 20:50:44.527712 7510 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:44.534014 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 7510
I20260128 20:50:44.534123 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:44.534415 4830 external_mini_cluster.cc:1468] Setting key f4c894d3ab011027a091a956c5352f09
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:44.543560 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:44.530055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:47179 (local address 127.4.183.190:46455)
0128 20:50:44.530247 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:44.530253 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:44.531147 (+ 894us) server_negotiation.cc:408] Connection header received
0128 20:50:44.532221 (+ 1074us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:44.532225 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:44.532313 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:44.532403 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:44.534485 (+ 2082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.535175 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:44.535934 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.536121 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:44.538938 (+ 2817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:44.538963 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:44.538965 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:44.538999 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:44.540888 (+ 1889us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:44.541679 (+ 791us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:44.541685 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:44.541688 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:44.541760 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:44.542161 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:44.542165 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:44.542167 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:44.542406 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:44.542527 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:44.543158 (+ 631us) server_negotiation.cc:300] Negotiation successful
0128 20:50:44.543320 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:50:44.544210 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:44.530469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:46455 (local address 127.4.183.130:47179)
0128 20:50:44.530975 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:44.531015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:44.531928 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:44.532605 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:44.532615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:44.533071 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:44.534272 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:44.534293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.535345 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:44.535349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:44.535800 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:44.535808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.536116 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:44.536845 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:44.536875 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:44.538716 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:44.541094 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:44.541101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:44.541114 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:44.541547 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:44.541901 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:44.541905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:44.541906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:44.542033 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:44.542551 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:44.542558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:44.542844 (+ 286us) client_negotiation.cc:769] Sending connection context
0128 20:50:44.543049 (+ 205us) client_negotiation.cc:240] Negotiation successful
0128 20:50:44.543320 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":125,"threads_started":1}
I20260128 20:50:44.545674 7639 heartbeater.cc:344] Connected to a master server at 127.4.183.190:46455
I20260128 20:50:44.545944 7639 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:44.546623 7639 heartbeater.cc:507] Master 127.4.183.190:46455 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:50:44.548189 7310 ts_manager.cc:194] Registered new tserver with Master: 440341c0331d4e77805544e7a2929199 (127.4.183.130:39809)
I20260128 20:50:44.548877 7310 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:47179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:44.593967 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:46455
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:44.713178 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:44.713528 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:44.713626 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:44.717641 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:44.717779 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:44.717906 7646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:44.723453 7646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:44.725317 7646 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:44.726671 7646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:44.735044 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:44.735044 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:44.735044 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:44.735483 7646 server_base.cc:1060] running on GCE node
I20260128 20:50:44.735940 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:44.736608 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:44.737804 7646 hybrid_clock.cc:648] HybridClock initialized: now 1769633444737774 us; error 52 us; skew 500 ppm
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:44.740921 7646 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:44.742259 7646 webserver.cc:492] Webserver started at http://127.4.183.131:33191/ using document root <none> and password file <none>
I20260128 20:50:44.742887 7646 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:44.742966 7646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:44.743212 7646 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:44.745203 7646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "055112fb765a460c8e3343a4abf3a090"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "15a15a68b8460f401b70bf3f1376d7ef"
server_key_iv: "55067d10dc0ccbf8713bd7cf07ee81e4"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.745793 7646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "055112fb765a460c8e3343a4abf3a090"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "15a15a68b8460f401b70bf3f1376d7ef"
server_key_iv: "55067d10dc0ccbf8713bd7cf07ee81e4"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.749809 7646 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:50:44.752943 7661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.755905 7646 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:44.756088 7646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "055112fb765a460c8e3343a4abf3a090"
format_stamp: "Formatted at 2026-01-28 20:50:44 on dist-test-slave-hss7"
server_key: "15a15a68b8460f401b70bf3f1376d7ef"
server_key_iv: "55067d10dc0ccbf8713bd7cf07ee81e4"
server_key_version: "encryptionkey@0"
I20260128 20:50:44.756217 7646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:44.771488 7646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:44.774758 7646 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:44.774993 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:44.775626 7646 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:44.776765 7646 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:44.776822 7646 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.776865 7646 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:44.776937 7646 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:44.787518 7646 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:39579
I20260128 20:50:44.787541 7774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:39579 every 8 connection(s)
I20260128 20:50:44.788599 7646 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:44.791289 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 7646
I20260128 20:50:44.791431 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:44.791774 4830 external_mini_cluster.cc:1468] Setting key 3f8b7042926c256a315a9515395cfdc5
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633444, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:44.802578 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:44.790497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:58257 (local address 127.4.183.190:46455)
0128 20:50:44.790641 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:44.790645 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:44.791596 (+ 951us) server_negotiation.cc:408] Connection header received
0128 20:50:44.792681 (+ 1085us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:44.792685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:44.792746 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:44.792854 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:44.794971 (+ 2117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.795512 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:44.796365 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.796541 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:44.799029 (+ 2488us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:44.799052 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:44.799054 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:44.799085 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:44.800398 (+ 1313us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:44.800977 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:44.800981 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:44.800983 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:44.801040 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:44.801392 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:44.801396 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:44.801398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:44.801574 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:44.801683 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:44.802256 (+ 573us) server_negotiation.cc:300] Negotiation successful
0128 20:50:44.802402 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260128 20:50:44.803239 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:44.790719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:46455 (local address 127.4.183.131:58257)
0128 20:50:44.791281 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:44.791334 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:44.792456 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:44.793037 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:44.793049 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:44.793687 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:44.794809 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:44.794826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.795651 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:44.795654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:44.796218 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:44.796227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:44.796447 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:44.797093 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:44.797123 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:44.798835 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:44.800559 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:44.800566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:44.800581 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:44.800860 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:44.801170 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:44.801173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:44.801178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:44.801285 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:44.801695 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:44.801705 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:44.801977 (+ 272us) client_negotiation.cc:769] Sending connection context
0128 20:50:44.802198 (+ 221us) client_negotiation.cc:240] Negotiation successful
0128 20:50:44.802505 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":114,"threads_started":1}
I20260128 20:50:44.804440 7775 heartbeater.cc:344] Connected to a master server at 127.4.183.190:46455
I20260128 20:50:44.804749 7775 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:44.805452 7775 heartbeater.cc:507] Master 127.4.183.190:46455 requested a full tablet report, sending...
I20260128 20:50:44.806576 7310 ts_manager.cc:194] Registered new tserver with Master: 055112fb765a460c8e3343a4abf3a090 (127.4.183.131:39579)
I20260128 20:50:44.807174 7310 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:58257
I20260128 20:50:44.818470 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 20:50:44.832181 7349 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:50:44.830623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.4.183.190:46455)
0128 20:50:44.830797 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:44.830801 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:44.830929 (+ 128us) server_negotiation.cc:408] Connection header received
0128 20:50:44.831046 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:44.831049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:44.831103 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:44.831193 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:44.832058 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57040: BlockingRecv error: recv got EOF from 127.0.0.1:57040 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260128 20:50:44.834087 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 7374
I20260128 20:50:44.840492 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 7510
I20260128 20:50:44.846570 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 7646
I20260128 20:50:44.853343 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 7278
2026-01-28T20:50:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3357 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7787](info): set up 2 sockets
Jan 28 20:50:44 dist-test-slave-hss7 krb5kdc[7787](info): commencing operation
krb5kdc: starting...
W20260128 20:50:46.886747 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.008s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:46 dist-test-slave-hss7 krb5kdc[7787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:47.048316 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:45199
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:47.168275 7803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:47.168625 7803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:47.168728 7803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:47.172370 7803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:47.172608 7803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:47.172730 7803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:47.172792 7803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:47.172847 7803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:47.177984 7803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:45199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:47.179471 7803 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:47.180526 7803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:47.187343 7808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:47.187358 7811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:47.187356 7809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:47.187726 7803 server_base.cc:1060] running on GCE node
I20260128 20:50:47.188301 7803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:47.189492 7803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:47.190726 7803 hybrid_clock.cc:648] HybridClock initialized: now 1769633447190714 us; error 32 us; skew 500 ppm
Jan 28 20:50:47 dist-test-slave-hss7 krb5kdc[7787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633447, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:47.194049 7803 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:47.195428 7803 webserver.cc:492] Webserver started at http://127.4.183.190:46209/ using document root <none> and password file <none>
I20260128 20:50:47.196070 7803 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:47.196156 7803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:47.196400 7803 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:47.198532 7803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "5270c6ea70c14c18aa1c5c51257531d0"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "5c657e0554ae9a40969bd97cac6d71de"
server_key_iv: "0fce1dde5161b164476dcefd81aa8168"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.199149 7803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "5270c6ea70c14c18aa1c5c51257531d0"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "5c657e0554ae9a40969bd97cac6d71de"
server_key_iv: "0fce1dde5161b164476dcefd81aa8168"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.203239 7803 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:50:47.205920 7818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.207083 7803 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260128 20:50:47.207263 7803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "5270c6ea70c14c18aa1c5c51257531d0"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "5c657e0554ae9a40969bd97cac6d71de"
server_key_iv: "0fce1dde5161b164476dcefd81aa8168"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.207392 7803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:47.224025 7803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:47.227595 7803 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:47.227823 7803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:47.237709 7803 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:45199
I20260128 20:50:47.237761 7870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:45199 every 8 connection(s)
I20260128 20:50:47.239078 7803 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:47.243495 7871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:47.245708 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 7803
I20260128 20:50:47.245900 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:47.246156 4830 external_mini_cluster.cc:1468] Setting key 764f542f7e84b06abcb1f35686475bf4
I20260128 20:50:47.250756 7871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0: Bootstrap starting.
I20260128 20:50:47.253448 7871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:47.254334 7871 log.cc:826] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0: Log is configured to *not* fsync() on all Append() calls
Jan 28 20:50:47 dist-test-slave-hss7 krb5kdc[7787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:47.257097 7871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0: No bootstrap required, opened a new log
I20260128 20:50:47.260332 7871 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5270c6ea70c14c18aa1c5c51257531d0" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45199 } }
I20260128 20:50:47.260624 7871 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:47.260712 7871 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5270c6ea70c14c18aa1c5c51257531d0, State: Initialized, Role: FOLLOWER
I20260128 20:50:47.261297 7871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5270c6ea70c14c18aa1c5c51257531d0" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45199 } }
I20260128 20:50:47.261497 7871 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:47.261577 7871 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:47.261691 7871 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:47.262010 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:47.247661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.4.183.190:45199)
0128 20:50:47.248198 (+ 537us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:47.248209 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:47.248249 (+ 40us) server_negotiation.cc:408] Connection header received
0128 20:50:47.248877 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:47.248898 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:47.249369 (+ 471us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:47.249774 (+ 405us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:47.250883 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.251697 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:47.252454 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.252788 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:47.255715 (+ 2927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:47.255740 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:47.255754 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:47.255801 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:47.258497 (+ 2696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:47.259305 (+ 808us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:47.259312 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:47.259319 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:47.259453 (+ 134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:47.259754 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:47.259757 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:47.259759 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:47.260179 (+ 420us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:47.260489 (+ 310us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:47.260691 (+ 202us) server_negotiation.cc:300] Negotiation successful
0128 20:50:47.261034 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":176,"threads_started":1}
I20260128 20:50:47.262959 7871 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5270c6ea70c14c18aa1c5c51257531d0" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45199 } }
I20260128 20:50:47.263412 7871 leader_election.cc:304] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5270c6ea70c14c18aa1c5c51257531d0; no voters:
I20260128 20:50:47.263782 7871 leader_election.cc:290] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:47.264185 7876 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:47.264798 7876 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5270c6ea70c14c18aa1c5c51257531d0, State: Running, Role: LEADER
I20260128 20:50:47.265190 7871 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:47.265156 7876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5270c6ea70c14c18aa1c5c51257531d0" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45199 } }
I20260128 20:50:47.267241 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5270c6ea70c14c18aa1c5c51257531d0. Latest consensus state: current_term: 1 leader_uuid: "5270c6ea70c14c18aa1c5c51257531d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5270c6ea70c14c18aa1c5c51257531d0" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45199 } } }
I20260128 20:50:47.267263 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5270c6ea70c14c18aa1c5c51257531d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5270c6ea70c14c18aa1c5c51257531d0" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45199 } } }
I20260128 20:50:47.267619 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:47.267863 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0 [sys.catalog]: This master's current role is: LEADER
W20260128 20:50:47.270906 7891 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:50:47.271026 7891 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:50:47.271102 7890 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:47.271896 7890 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:47.279877 7890 catalog_manager.cc:1357] Generated new cluster ID: de77b8167b1d47438437e461ac8bf7b8
I20260128 20:50:47.280028 7890 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:47.299155 7890 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:47.300249 7890 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:47.314021 7890 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5270c6ea70c14c18aa1c5c51257531d0: Generated new TSK 0
I20260128 20:50:47.315061 7890 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:47.384923 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:45199
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:47.510035 7899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:47.510385 7899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:47.510484 7899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:47.514355 7899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:47.514488 7899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:47.514619 7899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:47.519517 7899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:47.520866 7899 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:47.521924 7899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:47.529834 7905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:47.529821 7904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:47.529835 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:47.529981 7899 server_base.cc:1060] running on GCE node
I20260128 20:50:47.530542 7899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:47.531273 7899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:47.532487 7899 hybrid_clock.cc:648] HybridClock initialized: now 1769633447532455 us; error 54 us; skew 500 ppm
Jan 28 20:50:47 dist-test-slave-hss7 krb5kdc[7787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633447, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:47.535697 7899 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:47.537099 7899 webserver.cc:492] Webserver started at http://127.4.183.129:44437/ using document root <none> and password file <none>
I20260128 20:50:47.537873 7899 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:47.537941 7899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:47.538160 7899 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:47.540087 7899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "77574573688d4a7aa4b8acc1661c5f05"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "50435283ee17abd2a1a376e3c6c3219f"
server_key_iv: "afa82c22afe555b131e8d3dbd5be050c"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.540668 7899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "77574573688d4a7aa4b8acc1661c5f05"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "50435283ee17abd2a1a376e3c6c3219f"
server_key_iv: "afa82c22afe555b131e8d3dbd5be050c"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.544648 7899 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260128 20:50:47.547664 7914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.549556 7899 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:47.549778 7899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "77574573688d4a7aa4b8acc1661c5f05"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "50435283ee17abd2a1a376e3c6c3219f"
server_key_iv: "afa82c22afe555b131e8d3dbd5be050c"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.549918 7899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:47.577438 7899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:47.580701 7899 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:47.580924 7899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:47.581599 7899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:47.582582 7899 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:47.582648 7899 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.582722 7899 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:47.582768 7899 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.592881 7899 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:46535
I20260128 20:50:47.592904 8027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:46535 every 8 connection(s)
I20260128 20:50:47.594084 7899 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:50:47.603111 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 7899
I20260128 20:50:47.603416 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:47.603770 4830 external_mini_cluster.cc:1468] Setting key 7a6978a9c43d81f88b895cc9ece90bb5
Jan 28 20:50:47 dist-test-slave-hss7 krb5kdc[7787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633447, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:47.609015 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:47.596192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:45219 (local address 127.4.183.190:45199)
0128 20:50:47.596349 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:47.596353 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:47.597129 (+ 776us) server_negotiation.cc:408] Connection header received
0128 20:50:47.598073 (+ 944us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:47.598076 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:47.598130 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:47.598211 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:47.600083 (+ 1872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.600631 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:47.601379 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.601533 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:47.604669 (+ 3136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:47.604689 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:47.604692 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:47.604723 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:47.606628 (+ 1905us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:47.607358 (+ 730us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:47.607362 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:47.607363 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:47.607418 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:47.607757 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:47.607761 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:47.607763 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:47.607949 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:47.608060 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:47.608654 (+ 594us) server_negotiation.cc:300] Negotiation successful
0128 20:50:47.608831 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:50:47.609814 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:47.596434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45199 (local address 127.4.183.129:45219)
0128 20:50:47.596949 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:47.596989 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:47.597876 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:47.598405 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:47.598414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:47.598857 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:47.599875 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:47.599892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.600778 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:47.600781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:47.601249 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:47.601258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.601471 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:47.602151 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:47.602183 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:47.604472 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:47.606811 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:47.606818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:47.606830 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:47.607161 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:47.607527 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:47.607532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:47.607535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:47.607652 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:47.608061 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:47.608072 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:47.608374 (+ 302us) client_negotiation.cc:769] Sending connection context
0128 20:50:47.608617 (+ 243us) client_negotiation.cc:240] Negotiation successful
0128 20:50:47.608934 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260128 20:50:47.611318 8028 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45199
I20260128 20:50:47.611658 8028 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:47.612316 8028 heartbeater.cc:507] Master 127.4.183.190:45199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
I20260128 20:50:47.614315 7835 ts_manager.cc:194] Registered new tserver with Master: 77574573688d4a7aa4b8acc1661c5f05 (127.4.183.129:46535)
I20260128 20:50:47.615844 7835 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:45219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:47.661827 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:45199
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:47.782001 8035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:47.782277 8035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:47.782382 8035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:47.786161 8035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:47.786262 8035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:47.786394 8035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:47.791304 8035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:47.792551 8035 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:47.793561 8035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:47.801311 8040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:47.801335 8041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:47.801389 8035 server_base.cc:1060] running on GCE node
W20260128 20:50:47.801314 8043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:47.802029 8035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:47.802690 8035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:47.803913 8035 hybrid_clock.cc:648] HybridClock initialized: now 1769633447803898 us; error 33 us; skew 500 ppm
Jan 28 20:50:47 dist-test-slave-hss7 krb5kdc[7787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633447, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:47.807276 8035 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:47.808614 8035 webserver.cc:492] Webserver started at http://127.4.183.130:45623/ using document root <none> and password file <none>
I20260128 20:50:47.809296 8035 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:47.809374 8035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:47.809585 8035 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:47.811537 8035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "d8c06b2c79584921be5c786469167520"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "f5efd6357e84713e76b6aa03f14ed491"
server_key_iv: "8e2386b95637b1ce9cbf4a41faafb1ea"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.812391 8035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "d8c06b2c79584921be5c786469167520"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "f5efd6357e84713e76b6aa03f14ed491"
server_key_iv: "8e2386b95637b1ce9cbf4a41faafb1ea"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.816709 8035 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:50:47.820029 8050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.821758 8035 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260128 20:50:47.821964 8035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "d8c06b2c79584921be5c786469167520"
format_stamp: "Formatted at 2026-01-28 20:50:47 on dist-test-slave-hss7"
server_key: "f5efd6357e84713e76b6aa03f14ed491"
server_key_iv: "8e2386b95637b1ce9cbf4a41faafb1ea"
server_key_version: "encryptionkey@0"
I20260128 20:50:47.822098 8035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:47.845547 8035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:47.848825 8035 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:47.849042 8035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:47.849675 8035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:47.850612 8035 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:47.850660 8035 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.850704 8035 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:47.850719 8035 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:47.861591 8035 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:38027
I20260128 20:50:47.861605 8163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:38027 every 8 connection(s)
I20260128 20:50:47.862656 8035 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:47.869612 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8035
I20260128 20:50:47.869755 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:47.870011 4830 external_mini_cluster.cc:1468] Setting key dfc5fc1f54ae5b145c9c8029db64febb
Jan 28 20:50:47 dist-test-slave-hss7 krb5kdc[7787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633447, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:47.876713 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:47.864539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:51829 (local address 127.4.183.190:45199)
0128 20:50:47.864746 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:47.864751 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:47.865547 (+ 796us) server_negotiation.cc:408] Connection header received
0128 20:50:47.866458 (+ 911us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:47.866462 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:47.866550 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:47.866658 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:47.868675 (+ 2017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.869271 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:47.870080 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.870306 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:47.872815 (+ 2509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:47.872835 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:47.872838 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:47.872867 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:47.874479 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:47.875100 (+ 621us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:47.875104 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:47.875106 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:47.875164 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:47.875484 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:47.875490 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:47.875494 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:47.875677 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:47.875787 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:47.876372 (+ 585us) server_negotiation.cc:300] Negotiation successful
0128 20:50:47.876524 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260128 20:50:47.877528 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:47.864835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45199 (local address 127.4.183.130:51829)
0128 20:50:47.865373 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:47.865412 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:47.866243 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:47.866829 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:47.866838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:47.867362 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:47.868454 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:47.868484 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.869431 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:47.869435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:47.869933 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:47.869943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:47.870240 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:47.870964 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:47.870998 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:47.872622 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:47.874658 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:47.874665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:47.874681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:47.874981 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:47.875276 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:47.875283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:47.875285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:47.875388 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:47.875788 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:47.875799 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:47.876097 (+ 298us) client_negotiation.cc:769] Sending connection context
0128 20:50:47.876337 (+ 240us) client_negotiation.cc:240] Negotiation successful
0128 20:50:47.876661 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":131,"threads_started":1}
I20260128 20:50:47.878836 8164 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45199
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:50:47.879132 8164 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:47.879727 8164 heartbeater.cc:507] Master 127.4.183.190:45199 requested a full tablet report, sending...
I20260128 20:50:47.881024 7835 ts_manager.cc:194] Registered new tserver with Master: d8c06b2c79584921be5c786469167520 (127.4.183.130:38027)
I20260128 20:50:47.881732 7835 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:51829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:47.928925 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:45199
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:48.047284 8171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:48.047556 8171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:48.047658 8171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:48.051692 8171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:48.051797 8171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:48.051944 8171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:48.056813 8171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:48.058096 8171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:48.059057 8171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:48.066357 8177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:48.066460 8179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:48.066550 8171 server_base.cc:1060] running on GCE node
W20260128 20:50:48.066370 8176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:48.067150 8171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:48.067862 8171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:48.069053 8171 hybrid_clock.cc:648] HybridClock initialized: now 1769633448069024 us; error 48 us; skew 500 ppm
Jan 28 20:50:48 dist-test-slave-hss7 krb5kdc[7787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633448, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:48.072126 8171 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:48.073402 8171 webserver.cc:492] Webserver started at http://127.4.183.131:40967/ using document root <none> and password file <none>
I20260128 20:50:48.074040 8171 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:48.074119 8171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:48.074354 8171 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:48.076308 8171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "005f0aed35fa4c999d3fb703d071cd80"
format_stamp: "Formatted at 2026-01-28 20:50:48 on dist-test-slave-hss7"
server_key: "ecdd8c4269fe85b966f4729c0c8ffe52"
server_key_iv: "726f2130d7687e4db233bfd5be3c0159"
server_key_version: "encryptionkey@0"
I20260128 20:50:48.077257 8171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "005f0aed35fa4c999d3fb703d071cd80"
format_stamp: "Formatted at 2026-01-28 20:50:48 on dist-test-slave-hss7"
server_key: "ecdd8c4269fe85b966f4729c0c8ffe52"
server_key_iv: "726f2130d7687e4db233bfd5be3c0159"
server_key_version: "encryptionkey@0"
I20260128 20:50:48.081660 8171 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260128 20:50:48.084615 8186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:48.086088 8171 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:48.086277 8171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "005f0aed35fa4c999d3fb703d071cd80"
format_stamp: "Formatted at 2026-01-28 20:50:48 on dist-test-slave-hss7"
server_key: "ecdd8c4269fe85b966f4729c0c8ffe52"
server_key_iv: "726f2130d7687e4db233bfd5be3c0159"
server_key_version: "encryptionkey@0"
I20260128 20:50:48.086403 8171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:48.112493 8171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:48.116007 8171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:48.116379 8171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:48.117039 8171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:48.118072 8171 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:48.118126 8171 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:48.118199 8171 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:48.118237 8171 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:48.128757 8171 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:43813
I20260128 20:50:48.128907 8299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:43813 every 8 connection(s)
I20260128 20:50:48.129927 8171 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:48.136891 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8171
I20260128 20:50:48.137060 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:48.137389 4830 external_mini_cluster.cc:1468] Setting key c6f7a66843d4af934cde58b626a5d478
Jan 28 20:50:48 dist-test-slave-hss7 krb5kdc[7787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633448, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:48.143882 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:48.131925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:35581 (local address 127.4.183.190:45199)
0128 20:50:48.132118 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:48.132123 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:48.132914 (+ 791us) server_negotiation.cc:408] Connection header received
0128 20:50:48.133851 (+ 937us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:48.133854 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:48.133918 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:48.134023 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:48.135872 (+ 1849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:48.136386 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:48.137185 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:48.137416 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:48.140054 (+ 2638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:48.140070 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:48.140073 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:48.140104 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:48.141729 (+ 1625us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:48.142340 (+ 611us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:48.142343 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:48.142344 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:48.142388 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:48.142741 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:48.142743 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:48.142744 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:48.142905 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:48.142980 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:48.143596 (+ 616us) server_negotiation.cc:300] Negotiation successful
0128 20:50:48.143737 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260128 20:50:48.144611 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:48.132262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45199 (local address 127.4.183.131:35581)
0128 20:50:48.132727 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:48.132769 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:48.133638 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:48.134195 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:48.134204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:48.134673 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:48.135676 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:48.135694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:48.136523 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:48.136527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:48.136976 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:48.136984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:48.137309 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:48.137984 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:48.138028 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:48.139870 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:48.141888 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:48.141895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:48.141910 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:48.142235 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:48.142529 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:48.142532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:48.142534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:48.142645 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:48.143019 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:48.143026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:48.143309 (+ 283us) client_negotiation.cc:769] Sending connection context
0128 20:50:48.143545 (+ 236us) client_negotiation.cc:240] Negotiation successful
0128 20:50:48.143807 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260128 20:50:48.145913 8300 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45199
I20260128 20:50:48.146178 8300 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:48.146763 8300 heartbeater.cc:507] Master 127.4.183.190:45199 requested a full tablet report, sending...
I20260128 20:50:48.147948 7835 ts_manager.cc:194] Registered new tserver with Master: 005f0aed35fa4c999d3fb703d071cd80 (127.4.183.131:43813)
I20260128 20:50:48.148583 7835 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:35581
I20260128 20:50:48.151750 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 20:50:48.303798 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:50:48.298990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.4.183.190:45199)
0128 20:50:48.299191 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:48.299196 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:48.300398 (+ 1202us) server_negotiation.cc:408] Connection header received
0128 20:50:48.301693 (+ 1295us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:48.301698 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:48.301794 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:48.301935 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:48.303629 (+ 1694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54272: BlockingRecv error: recv got EOF from 127.0.0.1:54272 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260128 20:50:48.305753 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:50:48.304541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.4.183.190:45199)
0128 20:50:48.304693 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:48.304700 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:48.304788 (+ 88us) server_negotiation.cc:408] Connection header received
0128 20:50:48.304982 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:48.304985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:48.305040 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:48.305135 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:48.305625 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54274: BlockingRecv error: recv got EOF from 127.0.0.1:54274 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260128 20:50:48.307132 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:50:48.306148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.4.183.190:45199)
0128 20:50:48.306264 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:48.306267 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:48.306305 (+ 38us) server_negotiation.cc:408] Connection header received
0128 20:50:48.306436 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:48.306438 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:48.306483 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:48.306556 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:48.307003 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54288: BlockingRecv error: recv got EOF from 127.0.0.1:54288 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260128 20:50:48.312546 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:50:48.311159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.4.183.190:45199)
0128 20:50:48.311362 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:48.311368 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:48.311389 (+ 21us) server_negotiation.cc:408] Connection header received
0128 20:50:48.311715 (+ 326us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:48.311718 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:48.311778 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:48.311867 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:48.312419 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54292: BlockingRecv error: recv got EOF from 127.0.0.1:54292 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260128 20:50:48.322414 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 7899
I20260128 20:50:48.329061 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8035
I20260128 20:50:48.335497 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8171
I20260128 20:50:48.343325 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 7803
2026-01-28T20:50:48Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3490 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:48 dist-test-slave-hss7 krb5kdc[8323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:48 dist-test-slave-hss7 krb5kdc[8323](info): set up 2 sockets
Jan 28 20:50:48 dist-test-slave-hss7 krb5kdc[8323](info): commencing operation
krb5kdc: starting...
W20260128 20:50:50.382018 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.014s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:50 dist-test-slave-hss7 krb5kdc[8323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:50.545764 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:42959
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:50.678290 8339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:50.678583 8339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:50.678678 8339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:50.682248 8339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:50.682343 8339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:50.682365 8339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:50.682385 8339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:50.682402 8339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:50.687565 8339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:42959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:50.688912 8339 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:50.689905 8339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:50.696631 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:50.696619 8347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:50.696619 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:50.696782 8339 server_base.cc:1060] running on GCE node
I20260128 20:50:50.697620 8339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:50.698693 8339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:50.699888 8339 hybrid_clock.cc:648] HybridClock initialized: now 1769633450699860 us; error 46 us; skew 500 ppm
Jan 28 20:50:50 dist-test-slave-hss7 krb5kdc[8323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633450, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:50.703161 8339 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:50.704507 8339 webserver.cc:492] Webserver started at http://127.4.183.190:39069/ using document root <none> and password file <none>
I20260128 20:50:50.705089 8339 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:50.705209 8339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:50.705405 8339 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:50.707293 8339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "ec018bd052e5441c9fcd975091765f39"
format_stamp: "Formatted at 2026-01-28 20:50:50 on dist-test-slave-hss7"
server_key: "5b125d9d3049165f115e940d14c8c198"
server_key_iv: "79b00c6497f220f2c357fdfd65c6e283"
server_key_version: "encryptionkey@0"
I20260128 20:50:50.707795 8339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "ec018bd052e5441c9fcd975091765f39"
format_stamp: "Formatted at 2026-01-28 20:50:50 on dist-test-slave-hss7"
server_key: "5b125d9d3049165f115e940d14c8c198"
server_key_iv: "79b00c6497f220f2c357fdfd65c6e283"
server_key_version: "encryptionkey@0"
I20260128 20:50:50.711653 8339 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260128 20:50:50.714557 8354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:50.715839 8339 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:50:50.716013 8339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "ec018bd052e5441c9fcd975091765f39"
format_stamp: "Formatted at 2026-01-28 20:50:50 on dist-test-slave-hss7"
server_key: "5b125d9d3049165f115e940d14c8c198"
server_key_iv: "79b00c6497f220f2c357fdfd65c6e283"
server_key_version: "encryptionkey@0"
I20260128 20:50:50.716145 8339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:50.731962 8339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:50.745555 8339 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:50.745802 8339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:50.755069 8339 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:42959
I20260128 20:50:50.755069 8406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:42959 every 8 connection(s)
I20260128 20:50:50.756409 8339 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:50.759817 8407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:50.764350 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8339
I20260128 20:50:50.764532 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:50.764786 4830 external_mini_cluster.cc:1468] Setting key 713877b71a633c753b74be273ee2ebb2
I20260128 20:50:50.766903 8407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39: Bootstrap starting.
I20260128 20:50:50.770007 8407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:50.771011 8407 log.cc:826] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:50.773418 8407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39: No bootstrap required, opened a new log
Jan 28 20:50:50 dist-test-slave-hss7 krb5kdc[8323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:50.777328 8407 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec018bd052e5441c9fcd975091765f39" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 42959 } }
I20260128 20:50:50.777631 8407 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:50.777772 8407 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec018bd052e5441c9fcd975091765f39, State: Initialized, Role: FOLLOWER
I20260128 20:50:50.778339 8407 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec018bd052e5441c9fcd975091765f39" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 42959 } }
I20260128 20:50:50.778528 8407 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:50.778653 8407 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:50.778789 8407 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:50.780041 8407 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec018bd052e5441c9fcd975091765f39" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 42959 } }
I20260128 20:50:50.780269 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:50.766237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.4.183.190:42959)
0128 20:50:50.766755 (+ 518us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:50.766768 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:50.766814 (+ 46us) server_negotiation.cc:408] Connection header received
0128 20:50:50.767618 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:50.767646 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:50.768097 (+ 451us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:50.768538 (+ 441us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:50.769506 (+ 968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:50.770662 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:50.771508 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:50.771790 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:50.774471 (+ 2681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:50.774498 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:50.774513 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:50.774573 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:50.776843 (+ 2270us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:50.777477 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:50.777484 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:50.777489 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:50.777606 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:50.777919 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:50.777923 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:50.777925 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:50.778439 (+ 514us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:50.778687 (+ 248us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:50.779065 (+ 378us) server_negotiation.cc:300] Negotiation successful
0128 20:50:50.779379 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":153,"threads_started":1}
I20260128 20:50:50.780479 8407 leader_election.cc:304] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec018bd052e5441c9fcd975091765f39; no voters:
I20260128 20:50:50.780825 8407 leader_election.cc:290] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:50.781320 8412 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:50.781571 8412 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [term 1 LEADER]: Becoming Leader. State: Replica: ec018bd052e5441c9fcd975091765f39, State: Running, Role: LEADER
I20260128 20:50:50.781985 8412 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec018bd052e5441c9fcd975091765f39" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 42959 } }
I20260128 20:50:50.782227 8407 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:50.783689 8413 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec018bd052e5441c9fcd975091765f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec018bd052e5441c9fcd975091765f39" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 42959 } } }
I20260128 20:50:50.783838 8413 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:50.783931 4830 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 20:50:50.784335 8414 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec018bd052e5441c9fcd975091765f39. Latest consensus state: current_term: 1 leader_uuid: "ec018bd052e5441c9fcd975091765f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec018bd052e5441c9fcd975091765f39" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 42959 } } }
I20260128 20:50:50.784525 8414 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:50.784847 8421 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:50.788627 8421 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:50.797209 8421 catalog_manager.cc:1357] Generated new cluster ID: 4acac680d5dc4061a3779e60776f42b9
I20260128 20:50:50.797309 8421 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:50.805245 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:50.800537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.4.183.190:42959)
0128 20:50:50.800698 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:50.800704 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:50.800925 (+ 221us) server_negotiation.cc:408] Connection header received
0128 20:50:50.801096 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:50.801099 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:50.801197 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:50.801302 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:50.801998 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:50.802498 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:50.803180 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:50.803428 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:50.804065 (+ 637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:50.804096 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:50.804098 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:50:50.804643 (+ 545us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:50.804691 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:50:50.804710 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:50.804826 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:50.804939 (+ 113us) server_negotiation.cc:300] Negotiation successful
0128 20:50:50.805017 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260128 20:50:50.812376 8421 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:50.813511 8421 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:50.818909 8421 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ec018bd052e5441c9fcd975091765f39: Generated new TSK 0
I20260128 20:50:50.819748 8421 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:50:50.825819 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8339
2026-01-28T20:50:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2479 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:50 dist-test-slave-hss7 krb5kdc[8440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:50 dist-test-slave-hss7 krb5kdc[8440](info): set up 2 sockets
Jan 28 20:50:50 dist-test-slave-hss7 krb5kdc[8440](info): commencing operation
krb5kdc: starting...
W20260128 20:50:52.851198 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:52 dist-test-slave-hss7 krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:53.008853 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:33375
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:33375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:53.128057 8456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:53.128422 8456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:53.128530 8456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:53.132278 8456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:53.132422 8456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:53.132465 8456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:53.132508 8456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:53.132548 8456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:53.137904 8456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:33375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:33375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:53.139168 8456 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:53.140117 8456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:53.146720 8464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:53.146725 8462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:53.146720 8461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:53.147240 8456 server_base.cc:1060] running on GCE node
I20260128 20:50:53.147953 8456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:53.149061 8456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:53.150346 8456 hybrid_clock.cc:648] HybridClock initialized: now 1769633453150318 us; error 53 us; skew 500 ppm
Jan 28 20:50:53 dist-test-slave-hss7 krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633453, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:53.153863 8456 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:53.155149 8456 webserver.cc:492] Webserver started at http://127.4.183.190:33331/ using document root <none> and password file <none>
I20260128 20:50:53.155771 8456 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:53.155846 8456 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:53.156064 8456 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:53.158061 8456 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "d66cf848bd3d42688e8e4e214e24c176"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "5adf1550ca5ddbc5631fb6607485e436"
server_key_iv: "42182409f6d6f69a19110dbc451a2221"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.158639 8456 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "d66cf848bd3d42688e8e4e214e24c176"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "5adf1550ca5ddbc5631fb6607485e436"
server_key_iv: "42182409f6d6f69a19110dbc451a2221"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.163137 8456 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:50:53.166157 8471 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.167337 8456 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:53.167537 8456 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "d66cf848bd3d42688e8e4e214e24c176"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "5adf1550ca5ddbc5631fb6607485e436"
server_key_iv: "42182409f6d6f69a19110dbc451a2221"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.167667 8456 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:53.195726 8456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:53.199122 8456 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:53.199370 8456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:53.207530 8523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:33375 every 8 connection(s)
I20260128 20:50:53.207527 8456 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:33375
I20260128 20:50:53.208715 8456 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:53.211824 8524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:53.216712 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8456
I20260128 20:50:53.216863 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:53.217186 4830 external_mini_cluster.cc:1468] Setting key 70f53f7ae077f1ef49359c4a5eafce1c
I20260128 20:50:53.219282 8524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176: Bootstrap starting.
I20260128 20:50:53.221700 8524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:53.222644 8524 log.cc:826] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:53.224937 8524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176: No bootstrap required, opened a new log
Jan 28 20:50:53 dist-test-slave-hss7 krb5kdc[8440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:53.228274 8524 raft_consensus.cc:359] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66cf848bd3d42688e8e4e214e24c176" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33375 } }
I20260128 20:50:53.228540 8524 raft_consensus.cc:385] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:53.228636 8524 raft_consensus.cc:740] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66cf848bd3d42688e8e4e214e24c176, State: Initialized, Role: FOLLOWER
I20260128 20:50:53.229210 8524 consensus_queue.cc:260] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66cf848bd3d42688e8e4e214e24c176" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33375 } }
I20260128 20:50:53.229391 8524 raft_consensus.cc:399] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:53.229486 8524 raft_consensus.cc:493] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:53.229604 8524 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:53.230636 8524 raft_consensus.cc:515] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66cf848bd3d42688e8e4e214e24c176" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33375 } }
I20260128 20:50:53.231073 8524 leader_election.cc:304] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d66cf848bd3d42688e8e4e214e24c176; no voters:
I20260128 20:50:53.231453 8524 leader_election.cc:290] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:53.231894 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:53.218692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.4.183.190:33375)
0128 20:50:53.219208 (+ 516us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:53.219219 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:53.219258 (+ 39us) server_negotiation.cc:408] Connection header received
0128 20:50:53.219943 (+ 685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:53.219963 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:53.220322 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:53.220706 (+ 384us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:53.221770 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.222605 (+ 835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:53.223426 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.223744 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:53.226301 (+ 2557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:53.226327 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:53.226340 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:53.226390 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:53.228423 (+ 2033us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:53.228999 (+ 576us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:53.229004 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:53.229009 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:53.229104 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:53.229407 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:53.229412 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:53.229414 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:53.229919 (+ 505us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:53.230175 (+ 256us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:53.230447 (+ 272us) server_negotiation.cc:300] Negotiation successful
0128 20:50:53.230780 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":156,"threads_started":1}
I20260128 20:50:53.232141 8529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:53.232658 8529 raft_consensus.cc:697] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [term 1 LEADER]: Becoming Leader. State: Replica: d66cf848bd3d42688e8e4e214e24c176, State: Running, Role: LEADER
I20260128 20:50:53.233004 8524 sys_catalog.cc:565] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:53.233264 8529 consensus_queue.cc:237] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66cf848bd3d42688e8e4e214e24c176" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33375 } }
I20260128 20:50:53.235306 8531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d66cf848bd3d42688e8e4e214e24c176. Latest consensus state: current_term: 1 leader_uuid: "d66cf848bd3d42688e8e4e214e24c176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66cf848bd3d42688e8e4e214e24c176" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33375 } } }
I20260128 20:50:53.235469 8531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:53.235826 8530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d66cf848bd3d42688e8e4e214e24c176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66cf848bd3d42688e8e4e214e24c176" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33375 } } }
I20260128 20:50:53.235908 8530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176 [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:53.235901 8544 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:53.239712 8544 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:53.245923 8544 catalog_manager.cc:1357] Generated new cluster ID: 517eb4f30d8243b5aedd5542756c35ff
I20260128 20:50:53.246029 8544 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:53.266716 8544 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:53.268178 8544 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:53.282053 8544 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d66cf848bd3d42688e8e4e214e24c176: Generated new TSK 0
I20260128 20:50:53.283097 8544 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:53.341564 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:33375
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:53.478709 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:53.479001 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:53.479106 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:53.482815 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:53.482918 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:53.483040 8552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:53.487823 8552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:33375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:53.489075 8552 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:53.490082 8552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:53.497604 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:53.497601 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:53.497606 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:53.497928 8552 server_base.cc:1060] running on GCE node
I20260128 20:50:53.498364 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:53.499001 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:53.500182 8552 hybrid_clock.cc:648] HybridClock initialized: now 1769633453500148 us; error 56 us; skew 500 ppm
Jan 28 20:50:53 dist-test-slave-hss7 krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633453, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:53.503465 8552 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:53.504791 8552 webserver.cc:492] Webserver started at http://127.4.183.129:35337/ using document root <none> and password file <none>
I20260128 20:50:53.505512 8552 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:53.505575 8552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:53.505797 8552 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:53.507735 8552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "4cbf5c77aabb405cb45731f574ae18d2"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "88a67d9d32875f2362e81fbb1cadd854"
server_key_iv: "d2c96fa4ebdc8e5ceaf9fa19a519f885"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.508314 8552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "4cbf5c77aabb405cb45731f574ae18d2"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "88a67d9d32875f2362e81fbb1cadd854"
server_key_iv: "d2c96fa4ebdc8e5ceaf9fa19a519f885"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.512320 8552 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:50:53.516034 8567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.517617 8552 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260128 20:50:53.517798 8552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "4cbf5c77aabb405cb45731f574ae18d2"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "88a67d9d32875f2362e81fbb1cadd854"
server_key_iv: "d2c96fa4ebdc8e5ceaf9fa19a519f885"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.517933 8552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:53.542582 8552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:53.546000 8552 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:53.546237 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:53.546862 8552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:53.547830 8552 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:53.547881 8552 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.547946 8552 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:53.547977 8552 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.559244 8552 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:37689
I20260128 20:50:53.559260 8680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:37689 every 8 connection(s)
I20260128 20:50:53.560441 8552 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
Jan 28 20:50:53 dist-test-slave-hss7 krb5kdc[8440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633453, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:53.570627 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8552
I20260128 20:50:53.570777 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:53.571089 4830 external_mini_cluster.cc:1468] Setting key a28c57b718ad750948c235913687f27e
I20260128 20:50:53.575246 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:53.562599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:50881 (local address 127.4.183.190:33375)
0128 20:50:53.562770 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:53.562773 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:53.563582 (+ 809us) server_negotiation.cc:408] Connection header received
0128 20:50:53.564469 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:53.564472 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:53.564542 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:53.564631 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:53.566578 (+ 1947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.567099 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:53.567931 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.568158 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:53.570661 (+ 2503us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:53.570695 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:53.570700 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:53.570743 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:53.572453 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:53.573327 (+ 874us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:53.573330 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:53.573332 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:53.573385 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:53.573806 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:53.573808 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:53.573809 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:53.573979 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:53.574056 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:53.574951 (+ 895us) server_negotiation.cc:300] Negotiation successful
0128 20:50:53.575108 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260128 20:50:53.576040 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:53.562882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33375 (local address 127.4.183.129:50881)
0128 20:50:53.563403 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:53.563445 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:53.564237 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:53.564807 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:53.564816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:53.565289 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:53.566361 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:53.566379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.567226 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:53.567229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:53.567664 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:53.567672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.567870 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:53.568743 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:53.568772 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:53.570443 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:53.572680 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:53.572689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:53.572709 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:53.573174 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:53.573495 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:53.573501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:53.573506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:53.573694 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:53.574108 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:53.574121 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:53.574562 (+ 441us) client_negotiation.cc:769] Sending connection context
0128 20:50:53.574838 (+ 276us) client_negotiation.cc:240] Negotiation successful
0128 20:50:53.575181 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":119,"threads_started":1}
I20260128 20:50:53.577497 8681 heartbeater.cc:344] Connected to a master server at 127.4.183.190:33375
I20260128 20:50:53.577793 8681 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:53.578405 8681 heartbeater.cc:507] Master 127.4.183.190:33375 requested a full tablet report, sending...
I20260128 20:50:53.580233 8488 ts_manager.cc:194] Registered new tserver with Master: 4cbf5c77aabb405cb45731f574ae18d2 (127.4.183.129:37689)
I20260128 20:50:53.581637 8488 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:50881
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:53.638151 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:33375
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:53.756680 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:53.756963 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:53.757028 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:53.760704 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:53.760803 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:53.760926 8688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:53.765709 8688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:33375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:53.766897 8688 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:53.767794 8688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:53.775233 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:53.775234 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:53.775233 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:53.775605 8688 server_base.cc:1060] running on GCE node
I20260128 20:50:53.776036 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:53.776651 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:53.777853 8688 hybrid_clock.cc:648] HybridClock initialized: now 1769633453777816 us; error 50 us; skew 500 ppm
Jan 28 20:50:53 dist-test-slave-hss7 krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633453, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:53.784516 8688 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:53.785836 8688 webserver.cc:492] Webserver started at http://127.4.183.130:41673/ using document root <none> and password file <none>
I20260128 20:50:53.786448 8688 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:53.786540 8688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:53.786764 8688 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:53.788668 8688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "f6cfd63b3a9741ffa1794a866674b80d"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "33b87e34455fb73ab739ae2f24dd8c9a"
server_key_iv: "63737bcb1f3ee16f6b483185ec05fb72"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.789842 8688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "f6cfd63b3a9741ffa1794a866674b80d"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "33b87e34455fb73ab739ae2f24dd8c9a"
server_key_iv: "63737bcb1f3ee16f6b483185ec05fb72"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.794023 8688 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:50:53.796487 8703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.797672 8688 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 20:50:53.797811 8688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "f6cfd63b3a9741ffa1794a866674b80d"
format_stamp: "Formatted at 2026-01-28 20:50:53 on dist-test-slave-hss7"
server_key: "33b87e34455fb73ab739ae2f24dd8c9a"
server_key_iv: "63737bcb1f3ee16f6b483185ec05fb72"
server_key_version: "encryptionkey@0"
I20260128 20:50:53.797931 8688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:53.827350 8688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:53.831087 8688 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:53.831374 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:53.832065 8688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:53.833205 8688 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:53.833288 8688 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.833370 8688 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:53.833411 8688 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:53.845002 8688 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:42465
I20260128 20:50:53.845264 8816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:42465 every 8 connection(s)
I20260128 20:50:53.846295 8688 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:53.855984 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8688
I20260128 20:50:53.856132 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:53.856444 4830 external_mini_cluster.cc:1468] Setting key 1992541e6f759d109d1384050ef7a6b0
Jan 28 20:50:53 dist-test-slave-hss7 krb5kdc[8440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633453, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:53.863485 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:53.848878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:33955 (local address 127.4.183.190:33375)
0128 20:50:53.849120 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:53.849124 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:53.850221 (+ 1097us) server_negotiation.cc:408] Connection header received
0128 20:50:53.851254 (+ 1033us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:53.851258 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:53.851357 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:53.851456 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:53.853726 (+ 2270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.854281 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:53.855030 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.855185 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:53.858347 (+ 3162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:53.858370 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:53.858374 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:53.858412 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:53.860173 (+ 1761us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:53.860910 (+ 737us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:53.860916 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:53.860919 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:53.860995 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:53.861461 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:53.861467 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:53.861472 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:53.861728 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:53.861841 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:53.863027 (+ 1186us) server_negotiation.cc:300] Negotiation successful
0128 20:50:53.863217 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260128 20:50:53.864053 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:53.849402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33375 (local address 127.4.183.130:33955)
0128 20:50:53.850033 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:53.850076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:53.850982 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:53.851729 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:53.851740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:53.852235 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:53.853469 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:53.853491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.854445 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:53.854449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:53.854875 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:53.854884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:53.855111 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:53.855789 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:53.855834 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:53.858095 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:53.860419 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:53.860426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:53.860439 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:53.860769 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:53.861105 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:53.861108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:53.861110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:53.861340 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:53.861965 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:53.861976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:53.862659 (+ 683us) client_negotiation.cc:769] Sending connection context
0128 20:50:53.862909 (+ 250us) client_negotiation.cc:240] Negotiation successful
0128 20:50:53.863218 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":137,"threads_started":1}
I20260128 20:50:53.865938 8817 heartbeater.cc:344] Connected to a master server at 127.4.183.190:33375
I20260128 20:50:53.866513 8817 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:50:53.867244 8817 heartbeater.cc:507] Master 127.4.183.190:33375 requested a full tablet report, sending...
I20260128 20:50:53.868681 8488 ts_manager.cc:194] Registered new tserver with Master: f6cfd63b3a9741ffa1794a866674b80d (127.4.183.130:42465)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:50:53.869522 8488 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:33955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:53.918395 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:33375
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:54.051172 8824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:54.051752 8824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:54.051980 8824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:54.058840 8824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:54.059062 8824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:54.059324 8824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:54.064719 8824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:33375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:54.066023 8824 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:54.066959 8824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:54.074882 8829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:54.075083 8824 server_base.cc:1060] running on GCE node
W20260128 20:50:54.074882 8832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:54.074882 8830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:54.075906 8824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:54.076675 8824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:54.077888 8824 hybrid_clock.cc:648] HybridClock initialized: now 1769633454077865 us; error 40 us; skew 500 ppm
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633454, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:54.081986 8824 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:54.083387 8824 webserver.cc:492] Webserver started at http://127.4.183.131:40875/ using document root <none> and password file <none>
I20260128 20:50:54.084043 8824 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:54.084125 8824 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:54.084398 8824 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:54.086478 8824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "6ab7350121704595af379612859b6008"
format_stamp: "Formatted at 2026-01-28 20:50:54 on dist-test-slave-hss7"
server_key: "0ea84a62d0b1f0cfc822fb3e9af49fa4"
server_key_iv: "7b7bb1cde9d8a769cb1123f0de90ef40"
server_key_version: "encryptionkey@0"
I20260128 20:50:54.087041 8824 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "6ab7350121704595af379612859b6008"
format_stamp: "Formatted at 2026-01-28 20:50:54 on dist-test-slave-hss7"
server_key: "0ea84a62d0b1f0cfc822fb3e9af49fa4"
server_key_iv: "7b7bb1cde9d8a769cb1123f0de90ef40"
server_key_version: "encryptionkey@0"
I20260128 20:50:54.091080 8824 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:50:54.093859 8839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:54.095341 8824 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:54.095501 8824 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "6ab7350121704595af379612859b6008"
format_stamp: "Formatted at 2026-01-28 20:50:54 on dist-test-slave-hss7"
server_key: "0ea84a62d0b1f0cfc822fb3e9af49fa4"
server_key_iv: "7b7bb1cde9d8a769cb1123f0de90ef40"
server_key_version: "encryptionkey@0"
I20260128 20:50:54.095675 8824 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:54.111469 8824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:54.114944 8824 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:54.115211 8824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:54.115887 8824 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:54.116910 8824 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:54.116991 8824 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:54.117072 8824 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:54.117115 8824 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:54.128100 8824 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:46477
I20260128 20:50:54.128120 8952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:46477 every 8 connection(s)
I20260128 20:50:54.129281 8824 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:54.136792 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8824
I20260128 20:50:54.136970 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:54.137357 4830 external_mini_cluster.cc:1468] Setting key 24826048fa9bdae5e208d114b0deb58e
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633454, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:54.147711 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:54.132279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:58503 (local address 127.4.183.190:33375)
0128 20:50:54.132681 (+ 402us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:54.132687 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:54.134029 (+ 1342us) server_negotiation.cc:408] Connection header received
0128 20:50:54.135189 (+ 1160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:54.135194 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:54.135265 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:54.135361 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:54.137875 (+ 2514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:54.138642 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:54.139624 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:54.139854 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:54.143062 (+ 3208us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:54.143088 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:54.143090 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:54.143132 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:54.144739 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:54.145487 (+ 748us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:54.145491 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:54.145492 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:54.145550 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:54.145967 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:54.145971 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:54.145972 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:54.146169 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:54.146262 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:54.147258 (+ 996us) server_negotiation.cc:300] Negotiation successful
0128 20:50:54.147432 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260128 20:50:54.148893 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:54.133084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33375 (local address 127.4.183.131:58503)
0128 20:50:54.133880 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:54.133921 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:54.134934 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:54.135594 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:54.135605 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:54.136064 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:54.137599 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:54.137629 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:54.138847 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:54.138854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:54.139408 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:54.139421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:54.139722 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:54.140860 (+ 1138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:54.140902 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:54.142827 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:54.144918 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:54.144925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:54.144938 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:54.145355 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:54.145662 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:54.145665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:54.145667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:54.145796 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:54.146381 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:54.146393 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:54.146837 (+ 444us) client_negotiation.cc:769] Sending connection context
0128 20:50:54.147154 (+ 317us) client_negotiation.cc:240] Negotiation successful
0128 20:50:54.147611 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":145,"threads_started":1}
I20260128 20:50:54.150543 8953 heartbeater.cc:344] Connected to a master server at 127.4.183.190:33375
I20260128 20:50:54.150887 8953 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:54.151859 8953 heartbeater.cc:507] Master 127.4.183.190:33375 requested a full tablet report, sending...
I20260128 20:50:54.153791 8488 ts_manager.cc:194] Registered new tserver with Master: 6ab7350121704595af379612859b6008 (127.4.183.131:46477)
I20260128 20:50:54.154428 8488 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:58503
I20260128 20:50:54.166810 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633454, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633454, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:54.194713 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:54.184952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.4.183.190:33375)
0128 20:50:54.185239 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:54.185245 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:54.185414 (+ 169us) server_negotiation.cc:408] Connection header received
0128 20:50:54.185611 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:54.185614 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:54.185667 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:54.185760 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:54.186779 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:54.187302 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:54.187977 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:54.188188 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:54.190943 (+ 2755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:54.190969 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:54.190971 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:54.191012 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:54.192473 (+ 1461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:54.192978 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:54.192981 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:54.192982 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:54.193037 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:54.193428 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:54.193431 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:54.193433 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:54.193672 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:54.193794 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:54.194308 (+ 514us) server_negotiation.cc:300] Negotiation successful
0128 20:50:54.194528 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260128 20:50:54.195312 8488 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43046
I20260128 20:50:54.196640 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8552
I20260128 20:50:54.203035 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8688
I20260128 20:50:54.209038 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8824
I20260128 20:50:54.216893 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8456
2026-01-28T20:50:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3392 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8965](info): set up 2 sockets
Jan 28 20:50:54 dist-test-slave-hss7 krb5kdc[8965](info): commencing operation
krb5kdc: starting...
W20260128 20:50:56.260085 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.020s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:56 dist-test-slave-hss7 krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:56.426345 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:38743
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:38743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:56.544764 8981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:56.545050 8981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:56.545110 8981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:56.548750 8981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:56.548853 8981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:56.548877 8981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:56.548897 8981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:56.548915 8981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:56.553788 8981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:38743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:56.555102 8981 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:56.556075 8981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:56.562250 8986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:56.562295 8987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:56.562250 8989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:56.562695 8981 server_base.cc:1060] running on GCE node
I20260128 20:50:56.563243 8981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:56.564378 8981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:56.565623 8981 hybrid_clock.cc:648] HybridClock initialized: now 1769633456565595 us; error 51 us; skew 500 ppm
Jan 28 20:50:56 dist-test-slave-hss7 krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:56.568652 8981 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:56.570013 8981 webserver.cc:492] Webserver started at http://127.4.183.190:45227/ using document root <none> and password file <none>
I20260128 20:50:56.570639 8981 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:56.570698 8981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:56.570871 8981 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:56.572638 8981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "807ee596cf054801989f5e76933667bb"
format_stamp: "Formatted at 2026-01-28 20:50:56 on dist-test-slave-hss7"
server_key: "04ada461d26022d375171234a83a7af9"
server_key_iv: "4669f8392c74e425ee3c3b5c81e705ce"
server_key_version: "encryptionkey@0"
I20260128 20:50:56.573110 8981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "807ee596cf054801989f5e76933667bb"
format_stamp: "Formatted at 2026-01-28 20:50:56 on dist-test-slave-hss7"
server_key: "04ada461d26022d375171234a83a7af9"
server_key_iv: "4669f8392c74e425ee3c3b5c81e705ce"
server_key_version: "encryptionkey@0"
I20260128 20:50:56.577018 8981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:56.579821 8996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:56.581498 8981 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:56.581669 8981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "807ee596cf054801989f5e76933667bb"
format_stamp: "Formatted at 2026-01-28 20:50:56 on dist-test-slave-hss7"
server_key: "04ada461d26022d375171234a83a7af9"
server_key_iv: "4669f8392c74e425ee3c3b5c81e705ce"
server_key_version: "encryptionkey@0"
I20260128 20:50:56.581827 8981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:56.602254 8981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:56.610102 8981 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:56.610376 8981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:56.620229 8981 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:38743
I20260128 20:50:56.620229 9048 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:38743 every 8 connection(s)
I20260128 20:50:56.621879 8981 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:50:56.623713 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 8981
I20260128 20:50:56.623854 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:50:56.624120 4830 external_mini_cluster.cc:1468] Setting key 2e878e4bf84a08f95f3d381e821050d3
I20260128 20:50:56.625948 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:50:56.631834 9049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb: Bootstrap starting.
Jan 28 20:50:56 dist-test-slave-hss7 krb5kdc[8965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:56.634768 9049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb: Neither blocks nor log segments found. Creating new log.
I20260128 20:50:56.635732 9049 log.cc:826] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb: Log is configured to *not* fsync() on all Append() calls
I20260128 20:50:56.637957 9049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb: No bootstrap required, opened a new log
I20260128 20:50:56.639387 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:56.625897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.4.183.190:38743)
0128 20:50:56.626398 (+ 501us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:56.626423 (+ 25us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:56.626470 (+ 47us) server_negotiation.cc:408] Connection header received
0128 20:50:56.627772 (+ 1302us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:56.627797 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:56.628134 (+ 337us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:56.628540 (+ 406us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:56.629629 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:56.630477 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:56.631223 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:56.631548 (+ 325us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:56.634344 (+ 2796us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:56.634369 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:56.634383 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:56.634437 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:56.636455 (+ 2018us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:56.636993 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:56.636998 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:56.637003 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:56.637088 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:56.637452 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:56.637458 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:56.637463 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:56.637855 (+ 392us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:56.638101 (+ 246us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:56.638453 (+ 352us) server_negotiation.cc:300] Negotiation successful
0128 20:50:56.638732 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20260128 20:50:56.640877 9049 raft_consensus.cc:359] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807ee596cf054801989f5e76933667bb" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38743 } }
I20260128 20:50:56.641166 9049 raft_consensus.cc:385] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:50:56.641220 9049 raft_consensus.cc:740] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807ee596cf054801989f5e76933667bb, State: Initialized, Role: FOLLOWER
I20260128 20:50:56.641678 9049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807ee596cf054801989f5e76933667bb" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38743 } }
I20260128 20:50:56.641866 9049 raft_consensus.cc:399] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:50:56.641938 9049 raft_consensus.cc:493] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:50:56.642042 9049 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:50:56.642963 9049 raft_consensus.cc:515] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807ee596cf054801989f5e76933667bb" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38743 } }
I20260128 20:50:56.643301 9049 leader_election.cc:304] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807ee596cf054801989f5e76933667bb; no voters:
I20260128 20:50:56.643618 9049 leader_election.cc:290] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:50:56.643754 9054 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:50:56.643999 9054 raft_consensus.cc:697] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [term 1 LEADER]: Becoming Leader. State: Replica: 807ee596cf054801989f5e76933667bb, State: Running, Role: LEADER
I20260128 20:50:56.644369 9054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807ee596cf054801989f5e76933667bb" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38743 } }
I20260128 20:50:56.644951 9049 sys_catalog.cc:565] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:50:56.646649 9056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807ee596cf054801989f5e76933667bb. Latest consensus state: current_term: 1 leader_uuid: "807ee596cf054801989f5e76933667bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807ee596cf054801989f5e76933667bb" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38743 } } }
I20260128 20:50:56.646649 9055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807ee596cf054801989f5e76933667bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807ee596cf054801989f5e76933667bb" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 38743 } } }
I20260128 20:50:56.646845 9055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [sys.catalog]: This master's current role is: LEADER
I20260128 20:50:56.646845 9056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb [sys.catalog]: This master's current role is: LEADER
W20260128 20:50:56.650638 9069 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:50:56.650763 9069 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:50:56.650893 9067 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:50:56.651777 9067 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:50:56.658253 9067 catalog_manager.cc:1357] Generated new cluster ID: 6d5072c2da6f4132a5247293e1b2295f
I20260128 20:50:56.658367 9067 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:50:56.685873 9067 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:50:56.687333 9067 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:50:56.695791 9067 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 807ee596cf054801989f5e76933667bb: Generated new TSK 0
I20260128 20:50:56.696828 9067 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:56.760571 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:38743
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:56.899842 9077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:56.900110 9077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:56.900173 9077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:56.904255 9077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:56.904366 9077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:56.904506 9077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:50:56.909494 9077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:56.910782 9077 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:56.911697 9077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:56.919337 9082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:56.919323 9083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:56.919325 9085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:56.919656 9077 server_base.cc:1060] running on GCE node
I20260128 20:50:56.920097 9077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:56.920818 9077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:56.922164 9077 hybrid_clock.cc:648] HybridClock initialized: now 1769633456922104 us; error 56 us; skew 500 ppm
Jan 28 20:50:56 dist-test-slave-hss7 krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:56.926179 9077 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:50:56.927500 9077 webserver.cc:492] Webserver started at http://127.4.183.129:39173/ using document root <none> and password file <none>
I20260128 20:50:56.928097 9077 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:56.928150 9077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:56.928462 9077 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:56.930488 9077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "1e304ca8a4864378940dc263e9c7b020"
format_stamp: "Formatted at 2026-01-28 20:50:56 on dist-test-slave-hss7"
server_key: "c2c25d0039d57d059f7e5e7e6de3e8d0"
server_key_iv: "0c53b6b081affadbd7104fcec36951c8"
server_key_version: "encryptionkey@0"
I20260128 20:50:56.930999 9077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "1e304ca8a4864378940dc263e9c7b020"
format_stamp: "Formatted at 2026-01-28 20:50:56 on dist-test-slave-hss7"
server_key: "c2c25d0039d57d059f7e5e7e6de3e8d0"
server_key_iv: "0c53b6b081affadbd7104fcec36951c8"
server_key_version: "encryptionkey@0"
I20260128 20:50:56.934749 9077 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:50:56.937407 9092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:56.938601 9077 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:56.938733 9077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "1e304ca8a4864378940dc263e9c7b020"
format_stamp: "Formatted at 2026-01-28 20:50:56 on dist-test-slave-hss7"
server_key: "c2c25d0039d57d059f7e5e7e6de3e8d0"
server_key_iv: "0c53b6b081affadbd7104fcec36951c8"
server_key_version: "encryptionkey@0"
I20260128 20:50:56.938833 9077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:56.951522 9077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:56.954871 9077 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:56.955049 9077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:56.955574 9077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:56.956768 9077 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:56.956844 9077 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:56.956923 9077 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:56.956974 9077 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:56.967865 9077 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:40493
I20260128 20:50:56.967952 9205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:40493 every 8 connection(s)
I20260128 20:50:56.969050 9077 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
Jan 28 20:50:56 dist-test-slave-hss7 krb5kdc[8965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633456, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:56.979156 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9077
I20260128 20:50:56.979288 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:50:56.979579 4830 external_mini_cluster.cc:1468] Setting key e8e8772a13ff572fb554745447c9c2fa
I20260128 20:50:56.984807 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:56.971257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:47185 (local address 127.4.183.190:38743)
0128 20:50:56.971453 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:56.971457 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:56.972394 (+ 937us) server_negotiation.cc:408] Connection header received
0128 20:50:56.973437 (+ 1043us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:56.973441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:56.973509 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:56.973602 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:56.975596 (+ 1994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:56.976118 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:56.976852 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:56.977043 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:56.979745 (+ 2702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:56.979774 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:56.979777 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:56.979812 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:56.981465 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:56.982794 (+ 1329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:56.982798 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:56.982799 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:56.982861 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:56.983390 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:56.983392 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:56.983394 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:56.983574 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:56.983671 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:56.984492 (+ 821us) server_negotiation.cc:300] Negotiation successful
0128 20:50:56.984664 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260128 20:50:56.985885 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:56.971557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:38743 (local address 127.4.183.129:47185)
0128 20:50:56.972193 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:56.972240 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:56.973179 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:56.973810 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:56.973820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:56.974314 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:56.975359 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:56.975379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:56.976275 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:56.976284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:56.976712 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:56.976720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:56.977010 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:56.977781 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:56.977814 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:56.979528 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:56.981748 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:56.981775 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:56.981795 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:56.982633 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:56.982978 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:56.982984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:56.982988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:56.983179 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:56.983711 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:56.983723 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:56.984127 (+ 404us) client_negotiation.cc:769] Sending connection context
0128 20:50:56.984429 (+ 302us) client_negotiation.cc:240] Negotiation successful
0128 20:50:56.984799 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":163,"threads_started":1}
I20260128 20:50:56.987756 9206 heartbeater.cc:344] Connected to a master server at 127.4.183.190:38743
I20260128 20:50:56.988090 9206 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:56.988893 9206 heartbeater.cc:507] Master 127.4.183.190:38743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
I20260128 20:50:56.990967 9013 ts_manager.cc:194] Registered new tserver with Master: 1e304ca8a4864378940dc263e9c7b020 (127.4.183.129:40493)
I20260128 20:50:56.992405 9013 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:47185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:57.041121 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:38743
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:57.165951 9213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:57.166263 9213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:57.166374 9213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:57.170178 9213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:57.170301 9213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:57.170429 9213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:50:57.175298 9213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:57.176764 9213 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:57.177896 9213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:57.186501 9219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:57.187891 9221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:57.188092 9218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:57.188999 9213 server_base.cc:1060] running on GCE node
I20260128 20:50:57.189837 9213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:57.190807 9213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:57.192037 9213 hybrid_clock.cc:648] HybridClock initialized: now 1769633457191973 us; error 86 us; skew 500 ppm
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:57.196373 9213 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:50:57.197867 9213 webserver.cc:492] Webserver started at http://127.4.183.130:41913/ using document root <none> and password file <none>
I20260128 20:50:57.198688 9213 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:57.198794 9213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:57.199054 9213 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:57.201064 9213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "4b502d2070e74af7a35bdc0f69f8b291"
format_stamp: "Formatted at 2026-01-28 20:50:57 on dist-test-slave-hss7"
server_key: "4d19e28f5403364680d6a859b580f5d9"
server_key_iv: "e549d22e37c5e164d3f12b555c30fa06"
server_key_version: "encryptionkey@0"
I20260128 20:50:57.201711 9213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "4b502d2070e74af7a35bdc0f69f8b291"
format_stamp: "Formatted at 2026-01-28 20:50:57 on dist-test-slave-hss7"
server_key: "4d19e28f5403364680d6a859b580f5d9"
server_key_iv: "e549d22e37c5e164d3f12b555c30fa06"
server_key_version: "encryptionkey@0"
I20260128 20:50:57.205691 9213 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:50:57.208436 9228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:57.209836 9213 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:57.210003 9213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "4b502d2070e74af7a35bdc0f69f8b291"
format_stamp: "Formatted at 2026-01-28 20:50:57 on dist-test-slave-hss7"
server_key: "4d19e28f5403364680d6a859b580f5d9"
server_key_iv: "e549d22e37c5e164d3f12b555c30fa06"
server_key_version: "encryptionkey@0"
I20260128 20:50:57.210124 9213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:57.232285 9213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:57.235824 9213 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:57.236057 9213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:57.236675 9213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:57.237910 9213 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:57.237978 9213 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:57.238050 9213 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:57.238091 9213 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:57.251125 9213 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36401
I20260128 20:50:57.251202 9341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36401 every 8 connection(s)
I20260128 20:50:57.252297 9213 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:50:57.259155 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9213
I20260128 20:50:57.259267 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:50:57.259568 4830 external_mini_cluster.cc:1468] Setting key 6733c8a57e291c6caafc82739faadff3
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:57.269568 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:57.254375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:59595 (local address 127.4.183.190:38743)
0128 20:50:57.254527 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:57.254531 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:57.255354 (+ 823us) server_negotiation.cc:408] Connection header received
0128 20:50:57.256271 (+ 917us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:57.256276 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:57.256340 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:57.256433 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:57.258509 (+ 2076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.259056 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:57.259933 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.260121 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:57.263170 (+ 3049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:57.263193 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:57.263195 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:57.263227 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:57.267093 (+ 3866us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.267817 (+ 724us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.267823 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.267826 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.267889 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.268244 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.268248 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.268252 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.268460 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:57.268579 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:57.269203 (+ 624us) server_negotiation.cc:300] Negotiation successful
0128 20:50:57.269380 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:50:57.270332 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:57.254694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:38743 (local address 127.4.183.130:59595)
0128 20:50:57.255189 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:57.255228 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:57.256054 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:57.256636 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:57.256645 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:57.257205 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:57.258276 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:57.258297 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.259209 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:57.259213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:57.259790 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:57.259805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.260083 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:57.260985 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:57.261060 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:57.262969 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:57.267316 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:57.267324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:57.267338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:57.267689 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:57.268009 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:57.268012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:57.268014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:57.268129 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:57.268598 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:57.268607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:57.268897 (+ 290us) client_negotiation.cc:769] Sending connection context
0128 20:50:57.269134 (+ 237us) client_negotiation.cc:240] Negotiation successful
0128 20:50:57.269492 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260128 20:50:57.272152 9342 heartbeater.cc:344] Connected to a master server at 127.4.183.190:38743
I20260128 20:50:57.272591 9342 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:57.273321 9342 heartbeater.cc:507] Master 127.4.183.190:38743 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:50:57.274849 9013 ts_manager.cc:194] Registered new tserver with Master: 4b502d2070e74af7a35bdc0f69f8b291 (127.4.183.130:36401)
I20260128 20:50:57.275511 9013 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:59595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:57.318868 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:38743
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:57.439919 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:57.440193 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:57.440259 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:57.443984 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:57.444072 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:57.444161 9349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:50:57.449163 9349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:57.450405 9349 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:57.451321 9349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260128 20:50:57.459033 9349 server_base.cc:1060] running on GCE node
W20260128 20:50:57.458932 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:57.458935 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:57.458932 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:57.459905 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:57.460712 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:57.461930 9349 hybrid_clock.cc:648] HybridClock initialized: now 1769633457461903 us; error 49 us; skew 500 ppm
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:57.465196 9349 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:50:57.466531 9349 webserver.cc:492] Webserver started at http://127.4.183.131:44645/ using document root <none> and password file <none>
I20260128 20:50:57.467109 9349 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:57.467164 9349 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:57.467351 9349 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:57.469401 9349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "8e2791f4483e485dbb1872b67c23908b"
format_stamp: "Formatted at 2026-01-28 20:50:57 on dist-test-slave-hss7"
server_key: "502d469d8b1693fcd16359eef035e009"
server_key_iv: "8413695040646d6e5e0ed04e2e16bec1"
server_key_version: "encryptionkey@0"
I20260128 20:50:57.469918 9349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "8e2791f4483e485dbb1872b67c23908b"
format_stamp: "Formatted at 2026-01-28 20:50:57 on dist-test-slave-hss7"
server_key: "502d469d8b1693fcd16359eef035e009"
server_key_iv: "8413695040646d6e5e0ed04e2e16bec1"
server_key_version: "encryptionkey@0"
I20260128 20:50:57.473980 9349 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:50:57.476658 9364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:57.478042 9349 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:50:57.478174 9349 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "8e2791f4483e485dbb1872b67c23908b"
format_stamp: "Formatted at 2026-01-28 20:50:57 on dist-test-slave-hss7"
server_key: "502d469d8b1693fcd16359eef035e009"
server_key_iv: "8413695040646d6e5e0ed04e2e16bec1"
server_key_version: "encryptionkey@0"
I20260128 20:50:57.478281 9349 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:57.496750 9349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:57.500213 9349 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:57.500420 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:50:57.500991 9349 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:50:57.502141 9349 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:50:57.502195 9349 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:57.502240 9349 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:50:57.502255 9349 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:57.512854 9349 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:46283
I20260128 20:50:57.512941 9477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:46283 every 8 connection(s)
I20260128 20:50:57.514030 9349 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:50:57.516086 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9349
I20260128 20:50:57.516224 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:50:57.516497 4830 external_mini_cluster.cc:1468] Setting key 7a076cb7a13cb9d6fb4973c4da1fca23
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:57.531795 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:57.516238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:47565 (local address 127.4.183.190:38743)
0128 20:50:57.516395 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:57.516398 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:57.517401 (+ 1003us) server_negotiation.cc:408] Connection header received
0128 20:50:57.518883 (+ 1482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:57.518889 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:57.518998 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:57.519142 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:57.521333 (+ 2191us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.521918 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:57.523104 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.523445 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:57.526652 (+ 3207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:57.526689 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:57.526692 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:57.526746 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:57.528744 (+ 1998us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.529628 (+ 884us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.529632 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.529648 (+ 16us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.529735 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.530262 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.530265 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.530267 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.530540 (+ 273us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:57.530654 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:57.531440 (+ 786us) server_negotiation.cc:300] Negotiation successful
0128 20:50:57.531621 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260128 20:50:57.532792 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:57.516528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:38743 (local address 127.4.183.131:47565)
0128 20:50:57.517231 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0128 20:50:57.517273 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 20:50:57.518404 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:50:57.519370 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:50:57.519380 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:50:57.519854 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:50:57.520928 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:57.520949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.522251 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:50:57.522258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:50:57.522852 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:50:57.522868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.523217 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:57.524268 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:50:57.524340 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:50:57.526334 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:50:57.528950 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:57.528958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:57.528972 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:57.529448 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:57.529881 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:50:57.529888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:50:57.529890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:50:57.530053 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:50:57.530679 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:50:57.530688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:50:57.531074 (+ 386us) client_negotiation.cc:769] Sending connection context
0128 20:50:57.531331 (+ 257us) client_negotiation.cc:240] Negotiation successful
0128 20:50:57.531803 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":163,"threads_started":1}
I20260128 20:50:57.534583 9478 heartbeater.cc:344] Connected to a master server at 127.4.183.190:38743
I20260128 20:50:57.535019 9478 heartbeater.cc:461] Registering TS with master...
I20260128 20:50:57.535722 9478 heartbeater.cc:507] Master 127.4.183.190:38743 requested a full tablet report, sending...
I20260128 20:50:57.537235 9013 ts_manager.cc:194] Registered new tserver with Master: 8e2791f4483e485dbb1872b67c23908b (127.4.183.131:46283)
I20260128 20:50:57.537876 9013 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:47565
I20260128 20:50:57.544327 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:50:57.571717 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:57.561957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.4.183.129:40493)
0128 20:50:57.562338 (+ 381us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:57.562344 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:57.562369 (+ 25us) server_negotiation.cc:408] Connection header received
0128 20:50:57.562423 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:57.562428 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:57.562617 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:57.562743 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:57.563677 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.564393 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:57.565061 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.565449 (+ 388us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:57.567585 (+ 2136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:57.567616 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:57.567629 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:57.567657 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:57.569641 (+ 1984us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.570098 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.570102 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.570104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.570161 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.570626 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.570629 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.570630 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.570844 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:57.571040 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:57.571285 (+ 245us) server_negotiation.cc:300] Negotiation successful
0128 20:50:57.571452 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[8965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:50:57.583173 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:50:57.574542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.4.183.190:38743)
0128 20:50:57.574713 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0128 20:50:57.574719 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:50:57.574808 (+ 89us) server_negotiation.cc:408] Connection header received
0128 20:50:57.574977 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:50:57.574980 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:50:57.575043 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:50:57.575133 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:50:57.575901 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.576447 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:50:57.577088 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:50:57.577336 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:50:57.579682 (+ 2346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:50:57.579703 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:50:57.579706 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:50:57.579737 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:50:57.581407 (+ 1670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.581860 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.581863 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.581864 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.581914 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:50:57.582159 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:50:57.582163 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:50:57.582165 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:50:57.582384 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:50:57.582531 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0128 20:50:57.582829 (+ 298us) server_negotiation.cc:300] Negotiation successful
0128 20:50:57.582987 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260128 20:50:57.583617 9013 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52502
I20260128 20:50:57.584985 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9077
I20260128 20:50:57.592376 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9213
I20260128 20:50:57.598436 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9349
I20260128 20:50:57.605024 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 8981
2026-01-28T20:50:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3389 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[9490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[9490](info): set up 2 sockets
Jan 28 20:50:57 dist-test-slave-hss7 krb5kdc[9490](info): commencing operation
krb5kdc: starting...
W20260128 20:50:59.659557 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:50:59 dist-test-slave-hss7 krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:50:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:50:59.821848 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:43749
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:50:59.937865 9506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:50:59.938130 9506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:50:59.938194 9506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:50:59.941895 9506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:50:59.941987 9506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:50:59.942011 9506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:50:59.942031 9506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:50:59.942050 9506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:50:59.946784 9506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:50:59.948019 9506 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:50:59.948927 9506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:50:59.954831 9512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:59.954831 9511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:50:59.954823 9514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:50:59.954882 9506 server_base.cc:1060] running on GCE node
I20260128 20:50:59.955622 9506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:50:59.956580 9506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:50:59.957921 9506 hybrid_clock.cc:648] HybridClock initialized: now 1769633459957854 us; error 78 us; skew 500 ppm
Jan 28 20:50:59 dist-test-slave-hss7 krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:50:59.961055 9506 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:50:59.961966 9506 fs_manager.cc:362] Metadata directory not provided
I20260128 20:50:59.962076 9506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:50:59.962337 9506 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:50:59.964488 9506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "9c221ca578b04042a7c9c3bce82b7280"
format_stamp: "Formatted at 2026-01-28 20:50:59 on dist-test-slave-hss7"
server_key: "d3f92c441fa13841119f3339684a2ae4"
server_key_iv: "b15ea7f9f68e454067ea89ada5c6cc5e"
server_key_version: "encryptionkey@0"
I20260128 20:50:59.965087 9506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "9c221ca578b04042a7c9c3bce82b7280"
format_stamp: "Formatted at 2026-01-28 20:50:59 on dist-test-slave-hss7"
server_key: "d3f92c441fa13841119f3339684a2ae4"
server_key_iv: "b15ea7f9f68e454067ea89ada5c6cc5e"
server_key_version: "encryptionkey@0"
I20260128 20:50:59.969054 9506 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260128 20:50:59.971724 9520 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:50:59.972985 9506 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 20:50:59.973199 9506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "9c221ca578b04042a7c9c3bce82b7280"
format_stamp: "Formatted at 2026-01-28 20:50:59 on dist-test-slave-hss7"
server_key: "d3f92c441fa13841119f3339684a2ae4"
server_key_iv: "b15ea7f9f68e454067ea89ada5c6cc5e"
server_key_version: "encryptionkey@0"
I20260128 20:50:59.973340 9506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:50:59.987550 9506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:50:59.991102 9506 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:50:59.991355 9506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:00.000087 9572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:43749 every 8 connection(s)
I20260128 20:51:00.000097 9506 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:43749
I20260128 20:51:00.000912 9506 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:00.004055 9573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:00.009022 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9506
I20260128 20:51:00.009253 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:00.009578 4830 external_mini_cluster.cc:1468] Setting key f9d3066e358b126b3bb51913426000ce
I20260128 20:51:00.010540 9573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280: Bootstrap starting.
I20260128 20:51:00.013777 9573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:00.014822 9573 log.cc:826] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:00.017078 9573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280: No bootstrap required, opened a new log
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:00.021004 9573 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c221ca578b04042a7c9c3bce82b7280" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43749 } }
I20260128 20:51:00.021340 9573 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:00.021438 9573 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c221ca578b04042a7c9c3bce82b7280, State: Initialized, Role: FOLLOWER
I20260128 20:51:00.022002 9573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c221ca578b04042a7c9c3bce82b7280" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43749 } }
I20260128 20:51:00.022179 9573 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:00.022264 9573 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:00.022387 9573 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:00.023324 9573 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c221ca578b04042a7c9c3bce82b7280" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43749 } }
I20260128 20:51:00.023767 9573 leader_election.cc:304] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c221ca578b04042a7c9c3bce82b7280; no voters:
I20260128 20:51:00.024168 9573 leader_election.cc:290] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:00.024528 9578 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:00.024740 9578 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [term 1 LEADER]: Becoming Leader. State: Replica: 9c221ca578b04042a7c9c3bce82b7280, State: Running, Role: LEADER
I20260128 20:51:00.025061 9578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c221ca578b04042a7c9c3bce82b7280" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43749 } }
I20260128 20:51:00.025568 9573 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:00.026782 9580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c221ca578b04042a7c9c3bce82b7280. Latest consensus state: current_term: 1 leader_uuid: "9c221ca578b04042a7c9c3bce82b7280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c221ca578b04042a7c9c3bce82b7280" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43749 } } }
I20260128 20:51:00.026913 9580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:00.027274 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c221ca578b04042a7c9c3bce82b7280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c221ca578b04042a7c9c3bce82b7280" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43749 } } }
I20260128 20:51:00.027385 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:00.027540 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.011074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.4.183.190:43749)
0128 20:51:00.011566 (+ 492us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.011577 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.011616 (+ 39us) server_negotiation.cc:408] Connection header received
0128 20:51:00.012234 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.012253 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.012615 (+ 362us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.012962 (+ 347us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:00.014616 (+ 1654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.015646 (+ 1030us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.016459 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.016805 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.020021 (+ 3216us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:00.020065 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:00.020086 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:00.020163 (+ 77us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:00.022764 (+ 2601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.023574 (+ 810us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.023582 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.023590 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.023751 (+ 161us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.024129 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.024133 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.024137 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.025007 (+ 870us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:00.026084 (+ 1077us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.026328 (+ 244us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.026631 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":173,"threads_started":1}
I20260128 20:51:00.027716 9587 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:00.031486 9587 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:00.039256 9587 catalog_manager.cc:1357] Generated new cluster ID: df2000653e464d898a7b3e1b9b212832
I20260128 20:51:00.039363 9587 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:00.052628 9587 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:00.054070 9587 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:00.066584 9587 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280: Generated new TSK 0
I20260128 20:51:00.067638 9587 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:00.137835 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43749
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:00.266043 9601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:00.266340 9601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:00.266449 9601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:00.271060 9601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:00.271203 9601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:00.271332 9601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:00.277060 9601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:43749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:00.278530 9601 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:00.279486 9601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260128 20:51:00.287379 9601 server_base.cc:1060] running on GCE node
W20260128 20:51:00.287276 9607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:00.287276 9606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:00.287420 9609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:00.288337 9601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:00.289495 9601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:00.290735 9601 hybrid_clock.cc:648] HybridClock initialized: now 1769633460290698 us; error 57 us; skew 500 ppm
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:00.293849 9601 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:51:00.294700 9601 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:00.294767 9601 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:00.294950 9601 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:00.296864 9601 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "60a097b8e2c547a08a568c9a893969d8"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "5d17ca1495ce27a11d8bc4ff39fca8f5"
server_key_iv: "81e7824748fbefb4f331ed937d486403"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.297477 9601 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "60a097b8e2c547a08a568c9a893969d8"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "5d17ca1495ce27a11d8bc4ff39fca8f5"
server_key_iv: "81e7824748fbefb4f331ed937d486403"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.301561 9601 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260128 20:51:00.306854 9615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.308317 9601 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260128 20:51:00.308522 9601 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "60a097b8e2c547a08a568c9a893969d8"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "5d17ca1495ce27a11d8bc4ff39fca8f5"
server_key_iv: "81e7824748fbefb4f331ed937d486403"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.308681 9601 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:00.325342 9601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:00.329046 9601 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:00.329314 9601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:00.329957 9601 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:00.331174 9601 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:00.331269 9601 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.331358 9601 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:00.331405 9601 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.344388 9601 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:36317
I20260128 20:51:00.344455 9728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:36317 every 8 connection(s)
I20260128 20:51:00.345104 9601 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:00.346736 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9601
I20260128 20:51:00.346853 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:00.347113 4830 external_mini_cluster.cc:1468] Setting key 773de03ebfe40d8b37a1eed513d682df
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:51:00.362823 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.348722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:35491 (local address 127.4.183.190:43749)
0128 20:51:00.349001 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.349006 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.349025 (+ 19us) server_negotiation.cc:408] Connection header received
0128 20:51:00.349766 (+ 741us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.349770 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.349826 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.349895 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:00.352109 (+ 2214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.352666 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.353526 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.353727 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.357048 (+ 3321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:00.357076 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:00.357078 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:00.357112 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:00.359111 (+ 1999us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.360008 (+ 897us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.360011 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.360012 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.360070 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.360785 (+ 715us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.360791 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.360794 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.361225 (+ 431us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:00.361383 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.362194 (+ 811us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.362582 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260128 20:51:00.363888 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.347771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43749 (local address 127.4.183.129:35491)
0128 20:51:00.348456 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:00.348514 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:00.349543 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:00.350073 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:00.350082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:00.350641 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:00.351857 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.351880 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.352841 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.352846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:00.353375 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.353386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.353685 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.354477 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:00.354513 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:00.356803 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:00.359417 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:00.359426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:00.359443 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:00.359813 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:00.360254 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:00.360259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:00.360262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:00.360408 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:00.361404 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:00.361416 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:00.361828 (+ 412us) client_negotiation.cc:769] Sending connection context
0128 20:51:00.362128 (+ 300us) client_negotiation.cc:240] Negotiation successful
0128 20:51:00.362688 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":121,"threads_started":1}
I20260128 20:51:00.365948 9729 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43749
I20260128 20:51:00.366472 9729 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:00.367184 9729 heartbeater.cc:507] Master 127.4.183.190:43749 requested a full tablet report, sending...
I20260128 20:51:00.369297 9535 ts_manager.cc:194] Registered new tserver with Master: 60a097b8e2c547a08a568c9a893969d8 (127.4.183.129:36317)
I20260128 20:51:00.370793 9535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:35491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:00.409190 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43749
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:00.534442 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:00.535178 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:00.535367 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:00.539947 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:00.540076 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:00.540181 9736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:00.545053 9736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:43749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:00.546445 9736 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:00.547782 9736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:00.555980 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:00.556022 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:00.556146 9736 server_base.cc:1060] running on GCE node
W20260128 20:51:00.556146 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:00.556912 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:00.557963 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:00.559166 9736 hybrid_clock.cc:648] HybridClock initialized: now 1769633460559119 us; error 75 us; skew 500 ppm
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:00.562595 9736 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:51:00.563506 9736 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:00.563572 9736 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:00.563799 9736 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:00.565992 9736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "8120e117db8d4262a83a4309836525cd"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "cdef59be702b203140eb17e12f0824e1"
server_key_iv: "6ee92c399d41b3d7870a0d931a76cbc9"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.566635 9736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "8120e117db8d4262a83a4309836525cd"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "cdef59be702b203140eb17e12f0824e1"
server_key_iv: "6ee92c399d41b3d7870a0d931a76cbc9"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.570837 9736 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:00.573871 9750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.575959 9736 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260128 20:51:00.576236 9736 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "8120e117db8d4262a83a4309836525cd"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "cdef59be702b203140eb17e12f0824e1"
server_key_iv: "6ee92c399d41b3d7870a0d931a76cbc9"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.576606 9736 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:00.601807 9736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:00.606076 9736 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:00.606315 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:00.606981 9736 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:00.607975 9736 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:00.608053 9736 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.608129 9736 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:00.608178 9736 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.619402 9736 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:42171
I20260128 20:51:00.619401 9863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:42171 every 8 connection(s)
I20260128 20:51:00.620090 9736 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:00.627965 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9736
I20260128 20:51:00.628101 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:00.628361 4830 external_mini_cluster.cc:1468] Setting key e7c573945a010a1b6ac13dcb05220ecb
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:00.636739 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.622058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:49017 (local address 127.4.183.190:43749)
0128 20:51:00.622246 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.622250 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.623028 (+ 778us) server_negotiation.cc:408] Connection header received
0128 20:51:00.623960 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.623963 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.624029 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.624132 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:00.626008 (+ 1876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.626623 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.627354 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.627533 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.631496 (+ 3963us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:00.631647 (+ 151us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:00.631659 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:00.631706 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:00.633957 (+ 2251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.634790 (+ 833us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.634798 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.634801 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.634871 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.635230 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.635235 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.635239 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.635488 (+ 249us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:00.635616 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.636284 (+ 668us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.636456 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260128 20:51:00.637348 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.622330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43749 (local address 127.4.183.130:49017)
0128 20:51:00.622850 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:00.622890 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:00.623740 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:00.624303 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:00.624312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:00.624743 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:00.625813 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.625829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.626767 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.626775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:00.627220 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.627232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.627496 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.628265 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:00.628294 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:00.631221 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:00.634227 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:00.634236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:00.634249 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:00.634625 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:00.635003 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:00.635006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:00.635008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:00.635121 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:00.635651 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:00.635664 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:00.635966 (+ 302us) client_negotiation.cc:769] Sending connection context
0128 20:51:00.636171 (+ 205us) client_negotiation.cc:240] Negotiation successful
0128 20:51:00.636452 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":128,"threads_started":1}
I20260128 20:51:00.638789 9864 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43749
I20260128 20:51:00.639087 9864 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:51:00.639786 9864 heartbeater.cc:507] Master 127.4.183.190:43749 requested a full tablet report, sending...
I20260128 20:51:00.641347 9535 ts_manager.cc:194] Registered new tserver with Master: 8120e117db8d4262a83a4309836525cd (127.4.183.130:42171)
I20260128 20:51:00.642185 9535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:49017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:00.684383 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43749
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:00.804705 9871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:00.804994 9871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:00.805063 9871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:00.808766 9871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:00.808877 9871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:00.808969 9871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:00.814396 9871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:43749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:00.815788 9871 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:00.816891 9871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:00.824612 9879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:00.824811 9871 server_base.cc:1060] running on GCE node
W20260128 20:51:00.824612 9877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:00.824612 9876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:00.825435 9871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:00.826141 9871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:00.827369 9871 hybrid_clock.cc:648] HybridClock initialized: now 1769633460827340 us; error 35 us; skew 500 ppm
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:00.831287 9871 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:51:00.832288 9871 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:00.832401 9871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:00.832669 9871 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:00.834810 9871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "c9b7994f388844d19db8c7741249d3c2"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "3dff0b445c0942acc553d812c7164ff3"
server_key_iv: "dbdbe89c46ec89bb6e88525dbd8f0f02"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.835440 9871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "c9b7994f388844d19db8c7741249d3c2"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "3dff0b445c0942acc553d812c7164ff3"
server_key_iv: "dbdbe89c46ec89bb6e88525dbd8f0f02"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.839658 9871 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260128 20:51:00.842643 9885 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.843976 9871 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:00.844177 9871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "c9b7994f388844d19db8c7741249d3c2"
format_stamp: "Formatted at 2026-01-28 20:51:00 on dist-test-slave-hss7"
server_key: "3dff0b445c0942acc553d812c7164ff3"
server_key_iv: "dbdbe89c46ec89bb6e88525dbd8f0f02"
server_key_version: "encryptionkey@0"
I20260128 20:51:00.844318 9871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:00.866504 9871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:00.870015 9871 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:00.870260 9871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:00.870929 9871 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:00.871970 9871 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:00.872051 9871 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.872129 9871 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:00.872165 9871 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:00.883690 9871 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:38701
I20260128 20:51:00.883729 9998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:38701 every 8 connection(s)
I20260128 20:51:00.884348 9871 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:00.892717 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 9871
I20260128 20:51:00.892896 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:00.893194 4830 external_mini_cluster.cc:1468] Setting key 17d5216e76236886ef79f238ed3c65d9
Jan 28 20:51:00 dist-test-slave-hss7 krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:00.899924 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.887049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:59315 (local address 127.4.183.190:43749)
0128 20:51:00.887245 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.887249 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.888265 (+ 1016us) server_negotiation.cc:408] Connection header received
0128 20:51:00.889348 (+ 1083us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.889352 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.889420 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.889534 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:00.891496 (+ 1962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.892017 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.892775 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.892954 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.895813 (+ 2859us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:00.895831 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:00.895834 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:00.895871 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:00.897507 (+ 1636us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.898116 (+ 609us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.898119 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.898121 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.898170 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.898647 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.898650 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.898651 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.898838 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:00.898936 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.899613 (+ 677us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.899769 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260128 20:51:00.900701 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.887353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43749 (local address 127.4.183.131:59315)
0128 20:51:00.887986 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:00.888029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:00.889043 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:00.889745 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:00.889755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:00.890216 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:00.891264 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.891282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.892173 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.892184 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:00.892629 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.892637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.892919 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.893690 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:00.893724 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:00.895602 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:00.897682 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:00.897689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:00.897702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:00.898004 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:00.898350 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:00.898354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:00.898355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:00.898487 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:00.898974 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:00.898981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:00.899286 (+ 305us) client_negotiation.cc:769] Sending connection context
0128 20:51:00.899526 (+ 240us) client_negotiation.cc:240] Negotiation successful
0128 20:51:00.899802 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":151,"threads_started":1}
I20260128 20:51:00.902199 9999 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43749
I20260128 20:51:00.902580 9999 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:00.903192 9999 heartbeater.cc:507] Master 127.4.183.190:43749 requested a full tablet report, sending...
I20260128 20:51:00.904554 9535 ts_manager.cc:194] Registered new tserver with Master: c9b7994f388844d19db8c7741249d3c2 (127.4.183.131:38701)
I20260128 20:51:00.905274 9535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:59315
I20260128 20:51:00.908435 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 20:51:00.918751 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.910874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.4.183.190:43749)
0128 20:51:00.911157 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.911164 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.911348 (+ 184us) server_negotiation.cc:408] Connection header received
0128 20:51:00.911533 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.911543 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.911592 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.911700 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:00.912611 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.913122 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.913965 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.914119 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.915439 (+ 1320us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:00.915471 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:00.915474 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:00.915503 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:00.916893 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.917504 (+ 611us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.917507 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.917509 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.917564 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:00.917855 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:00.917858 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:00.917859 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:00.918065 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:00.918181 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.918492 (+ 311us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.918616 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260128 20:51:00.922472 9535 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:00.925292 9535 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:00.940390 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.935701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:42171 (local address 127.0.0.1:50892)
0128 20:51:00.936121 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:00.936169 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:00.936332 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:00.937090 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:00.937097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:00.937157 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:00.937544 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.937555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.938868 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.938871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:00.939808 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.939816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.939916 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.939999 (+ 83us) client_negotiation.cc:769] Sending connection context
0128 20:51:00.940102 (+ 103us) client_negotiation.cc:240] Negotiation successful
0128 20:51:00.940209 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260128 20:51:00.941237 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.936334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:38701 (local address 127.0.0.1:42630)
0128 20:51:00.937079 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:00.937094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:00.937234 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:00.937949 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:00.937954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:00.937971 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:00.938198 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.938205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.939672 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.939676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:00.940505 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.940517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.940911 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.940934 (+ 23us) client_negotiation.cc:769] Sending connection context
0128 20:51:00.940990 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:51:00.941061 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":114,"threads_started":1}
I20260128 20:51:00.941359 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.935794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.4.183.130:42171)
0128 20:51:00.936220 (+ 426us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.936227 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.936254 (+ 27us) server_negotiation.cc:408] Connection header received
0128 20:51:00.936541 (+ 287us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.936547 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.936757 (+ 210us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.936912 (+ 155us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:00.937718 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.938729 (+ 1011us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.940169 (+ 1440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.940819 (+ 650us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.940959 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.941038 (+ 79us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.941188 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":169,"threads_started":1}
I20260128 20:51:00.941933 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.937464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:36317 (local address 127.0.0.1:34504)
0128 20:51:00.937782 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:00.937795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:00.937870 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:00.938299 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:00.938302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:00.938323 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:00.938595 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.938603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.939862 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.939868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:00.940961 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:00.940974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.941089 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.941106 (+ 17us) client_negotiation.cc:769] Sending connection context
0128 20:51:00.941744 (+ 638us) client_negotiation.cc:240] Negotiation successful
0128 20:51:00.941815 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260128 20:51:00.941984 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.936533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.4.183.131:38701)
0128 20:51:00.936924 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.936930 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.937221 (+ 291us) server_negotiation.cc:408] Connection header received
0128 20:51:00.937383 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.937392 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.937608 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.937776 (+ 168us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:00.938372 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.939518 (+ 1146us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.940927 (+ 1409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.941549 (+ 622us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.941681 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.941755 (+ 74us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.941851 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
I20260128 20:51:00.942709 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:00.937492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.4.183.129:36317)
0128 20:51:00.937883 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:00.937890 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:00.937912 (+ 22us) server_negotiation.cc:408] Connection header received
0128 20:51:00.937988 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:00.937993 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:00.938143 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:00.938283 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:00.938729 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.939721 (+ 992us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:00.941286 (+ 1565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:00.942140 (+ 854us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:00.942260 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:00.942334 (+ 74us) server_negotiation.cc:300] Negotiation successful
0128 20:51:00.942525 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260128 20:51:00.945128 9933 tablet_service.cc:1505] Processing CreateTablet for tablet 39ac60c98be0420ab04316ba77a467b4 (DEFAULT_TABLE table=test-table [id=3fe87f271bfd4a108ca09835bb6bec05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:00.945128 9798 tablet_service.cc:1505] Processing CreateTablet for tablet 39ac60c98be0420ab04316ba77a467b4 (DEFAULT_TABLE table=test-table [id=3fe87f271bfd4a108ca09835bb6bec05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:00.945511 9663 tablet_service.cc:1505] Processing CreateTablet for tablet 39ac60c98be0420ab04316ba77a467b4 (DEFAULT_TABLE table=test-table [id=3fe87f271bfd4a108ca09835bb6bec05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:00.946393 9798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ac60c98be0420ab04316ba77a467b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:00.946393 9933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ac60c98be0420ab04316ba77a467b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:00.946434 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ac60c98be0420ab04316ba77a467b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:00.953990 10018 tablet_bootstrap.cc:492] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Bootstrap starting.
I20260128 20:51:00.955396 10017 tablet_bootstrap.cc:492] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Bootstrap starting.
I20260128 20:51:00.956699 10018 tablet_bootstrap.cc:654] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:00.957590 10017 tablet_bootstrap.cc:654] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:00.957631 10016 tablet_bootstrap.cc:492] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: Bootstrap starting.
I20260128 20:51:00.958194 10018 log.cc:826] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:00.958631 10017 log.cc:826] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:00.960148 10016 tablet_bootstrap.cc:654] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:00.961062 10018 tablet_bootstrap.cc:492] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: No bootstrap required, opened a new log
I20260128 20:51:00.961118 10017 tablet_bootstrap.cc:492] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: No bootstrap required, opened a new log
I20260128 20:51:00.961081 10016 log.cc:826] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:00.961501 10017 ts_tablet_manager.cc:1403] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 20:51:00.961540 10018 ts_tablet_manager.cc:1403] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260128 20:51:00.963804 10016 tablet_bootstrap.cc:492] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: No bootstrap required, opened a new log
I20260128 20:51:00.964121 10016 ts_tablet_manager.cc:1403] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260128 20:51:00.966035 10018 raft_consensus.cc:359] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:00.965868 10017 raft_consensus.cc:359] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:00.966391 10018 raft_consensus.cc:385] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:00.966398 10017 raft_consensus.cc:385] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:00.966465 10018 raft_consensus.cc:740] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b7994f388844d19db8c7741249d3c2, State: Initialized, Role: FOLLOWER
I20260128 20:51:00.966477 10017 raft_consensus.cc:740] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8120e117db8d4262a83a4309836525cd, State: Initialized, Role: FOLLOWER
I20260128 20:51:00.967077 10018 consensus_queue.cc:260] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:00.967633 10017 consensus_queue.cc:260] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:00.967993 10016 raft_consensus.cc:359] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:00.968255 10016 raft_consensus.cc:385] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:00.968273 10018 ts_tablet_manager.cc:1434] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260128 20:51:00.968336 10016 raft_consensus.cc:740] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a097b8e2c547a08a568c9a893969d8, State: Initialized, Role: FOLLOWER
I20260128 20:51:00.968854 9999 heartbeater.cc:499] Master 127.4.183.190:43749 was elected leader, sending a full tablet report...
I20260128 20:51:00.968957 10016 consensus_queue.cc:260] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:00.969238 10017 ts_tablet_manager.cc:1434] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260128 20:51:00.969806 9864 heartbeater.cc:499] Master 127.4.183.190:43749 was elected leader, sending a full tablet report...
I20260128 20:51:00.970356 9729 heartbeater.cc:499] Master 127.4.183.190:43749 was elected leader, sending a full tablet report...
I20260128 20:51:00.970829 10016 ts_tablet_manager.cc:1434] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260128 20:51:01.008127 10024 raft_consensus.cc:493] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:01.008385 10024 raft_consensus.cc:515] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:01.009665 10024 leader_election.cc:290] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8120e117db8d4262a83a4309836525cd (127.4.183.130:42171), c9b7994f388844d19db8c7741249d3c2 (127.4.183.131:38701)
I20260128 20:51:01.014066 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.009914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:38701 (local address 127.4.183.129:50957)
0128 20:51:01.010467 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:01.010482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:01.010636 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:01.010977 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:01.010980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:01.011000 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:01.011684 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:01.011695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.012538 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:01.012543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:01.013532 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:01.013544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.013712 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:01.013737 (+ 25us) client_negotiation.cc:769] Sending connection context
0128 20:51:01.013792 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 20:51:01.013868 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":192,"threads_started":1}
I20260128 20:51:01.014755 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.010438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:53483 (local address 127.4.183.130:42171)
0128 20:51:01.010572 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.010577 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.010595 (+ 18us) server_negotiation.cc:408] Connection header received
0128 20:51:01.011081 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.011084 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.011137 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.011218 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:01.011841 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.012619 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:01.013967 (+ 1348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.014499 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:01.014533 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.014582 (+ 49us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.014649 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260128 20:51:01.014739 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.010035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:50957 (local address 127.4.183.131:38701)
0128 20:51:01.010244 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.010251 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.010571 (+ 320us) server_negotiation.cc:408] Connection header received
0128 20:51:01.010775 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.010780 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.010843 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.010939 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:01.011890 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.012396 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:01.013688 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.014250 (+ 562us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:01.014297 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.014363 (+ 66us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.014435 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260128 20:51:01.015106 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.009930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:42171 (local address 127.4.183.129:53483)
0128 20:51:01.010495 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:01.010511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:01.010625 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:01.011268 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:01.011272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:01.011292 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:01.011684 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:01.011695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.012736 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:01.012739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:01.013565 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:01.013571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.013710 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:01.013737 (+ 27us) client_negotiation.cc:769] Sending connection context
0128 20:51:01.013792 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 20:51:01.013868 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":167,"threads_started":1}
I20260128 20:51:01.015573 9953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39ac60c98be0420ab04316ba77a467b4" candidate_uuid: "60a097b8e2c547a08a568c9a893969d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b7994f388844d19db8c7741249d3c2" is_pre_election: true
I20260128 20:51:01.015882 9953 raft_consensus.cc:2468] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60a097b8e2c547a08a568c9a893969d8 in term 0.
I20260128 20:51:01.016840 9619 leader_election.cc:304] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60a097b8e2c547a08a568c9a893969d8, c9b7994f388844d19db8c7741249d3c2; no voters:
I20260128 20:51:01.016942 9818 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39ac60c98be0420ab04316ba77a467b4" candidate_uuid: "60a097b8e2c547a08a568c9a893969d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8120e117db8d4262a83a4309836525cd" is_pre_election: true
I20260128 20:51:01.017288 9818 raft_consensus.cc:2468] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60a097b8e2c547a08a568c9a893969d8 in term 0.
I20260128 20:51:01.017283 10024 raft_consensus.cc:2804] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:01.017390 10024 raft_consensus.cc:493] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:01.017443 10024 raft_consensus.cc:3060] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:01.018756 10024 raft_consensus.cc:515] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:01.019234 10024 leader_election.cc:290] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8120e117db8d4262a83a4309836525cd (127.4.183.130:42171), c9b7994f388844d19db8c7741249d3c2 (127.4.183.131:38701)
I20260128 20:51:01.019883 9953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39ac60c98be0420ab04316ba77a467b4" candidate_uuid: "60a097b8e2c547a08a568c9a893969d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b7994f388844d19db8c7741249d3c2"
I20260128 20:51:01.019883 9818 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39ac60c98be0420ab04316ba77a467b4" candidate_uuid: "60a097b8e2c547a08a568c9a893969d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8120e117db8d4262a83a4309836525cd"
I20260128 20:51:01.020045 9818 raft_consensus.cc:3060] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:01.020045 9953 raft_consensus.cc:3060] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:01.021503 9818 raft_consensus.cc:2468] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60a097b8e2c547a08a568c9a893969d8 in term 1.
I20260128 20:51:01.021503 9953 raft_consensus.cc:2468] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60a097b8e2c547a08a568c9a893969d8 in term 1.
I20260128 20:51:01.022001 9619 leader_election.cc:304] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60a097b8e2c547a08a568c9a893969d8, c9b7994f388844d19db8c7741249d3c2; no voters:
I20260128 20:51:01.022270 10024 raft_consensus.cc:2804] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:01.022552 10024 raft_consensus.cc:697] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [term 1 LEADER]: Becoming Leader. State: Replica: 60a097b8e2c547a08a568c9a893969d8, State: Running, Role: LEADER
I20260128 20:51:01.022976 10024 consensus_queue.cc:237] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } }
I20260128 20:51:01.026798 9535 catalog_manager.cc:5654] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60a097b8e2c547a08a568c9a893969d8 (127.4.183.129). New cstate: current_term: 1 leader_uuid: "60a097b8e2c547a08a568c9a893969d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60a097b8e2c547a08a568c9a893969d8" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 36317 } health_report { overall_health: HEALTHY } } }
I20260128 20:51:01.048223 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.044752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.4.183.129:36317)
0128 20:51:01.044905 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.044908 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.044923 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:01.045010 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.045013 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.045072 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.045249 (+ 177us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:01.045599 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.046129 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:01.046854 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:01.047012 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:01.047120 (+ 108us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:01.047645 (+ 525us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:01.047759 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.047947 (+ 188us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.048012 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260128 20:51:01.055784 9818 raft_consensus.cc:1275] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 1 FOLLOWER]: Refusing update from remote peer 60a097b8e2c547a08a568c9a893969d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:01.055783 9953 raft_consensus.cc:1275] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 1 FOLLOWER]: Refusing update from remote peer 60a097b8e2c547a08a568c9a893969d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:01.056825 10024 consensus_queue.cc:1048] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8120e117db8d4262a83a4309836525cd" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:01.057109 10027 consensus_queue.cc:1048] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9b7994f388844d19db8c7741249d3c2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:01.066555 10032 mvcc.cc:204] Tried to move back new op lower bound from 7248418656475049984 to 7248418656353693696. Current Snapshot: MvccSnapshot[applied={T|T < 7248418656475049984}]
I20260128 20:51:01.077764 9537 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:01.077992 9537 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:01.081332 9537 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9c221ca578b04042a7c9c3bce82b7280: Sending DeleteTablet for 3 replicas of tablet 39ac60c98be0420ab04316ba77a467b4
I20260128 20:51:01.082587 9798 tablet_service.cc:1552] Processing DeleteTablet for tablet 39ac60c98be0420ab04316ba77a467b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:01 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:50892
I20260128 20:51:01.082671 9663 tablet_service.cc:1552] Processing DeleteTablet for tablet 39ac60c98be0420ab04316ba77a467b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:01 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:34504
I20260128 20:51:01.082839 9933 tablet_service.cc:1552] Processing DeleteTablet for tablet 39ac60c98be0420ab04316ba77a467b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:01 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:42630
I20260128 20:51:01.083487 10042 tablet_replica.cc:333] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: stopping tablet replica
I20260128 20:51:01.083808 10040 tablet_replica.cc:333] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: stopping tablet replica
I20260128 20:51:01.083834 10042 raft_consensus.cc:2243] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:01.084133 10041 tablet_replica.cc:333] T 39ac60c98be0420ab04316ba77a467b4 P 60a097b8e2c547a08a568c9a893969d8: stopping tablet replica
I20260128 20:51:01.084314 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9601
I20260128 20:51:01.086077 10040 raft_consensus.cc:2243] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:01.086571 10040 raft_consensus.cc:2272] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:01.088645 10040 ts_tablet_manager.cc:1916] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:01.084280 10042 raft_consensus.cc:2272] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:01.091097 10042 ts_tablet_manager.cc:1916] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:01.092032 10040 ts_tablet_manager.cc:1929] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:01.092154 10040 log.cc:1199] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/39ac60c98be0420ab04316ba77a467b4
I20260128 20:51:01.092526 10040 ts_tablet_manager.cc:1950] T 39ac60c98be0420ab04316ba77a467b4 P 8120e117db8d4262a83a4309836525cd: Deleting consensus metadata
I20260128 20:51:01.094039 10042 ts_tablet_manager.cc:1929] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:01.094182 10042 log.cc:1199] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/39ac60c98be0420ab04316ba77a467b4
I20260128 20:51:01.094218 9522 catalog_manager.cc:4985] TS 8120e117db8d4262a83a4309836525cd (127.4.183.130:42171): tablet 39ac60c98be0420ab04316ba77a467b4 (table test-table [id=3fe87f271bfd4a108ca09835bb6bec05]) successfully deleted
W20260128 20:51:01.094527 9522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.183.129:36317 (error 108)
I20260128 20:51:01.094555 10042 ts_tablet_manager.cc:1950] T 39ac60c98be0420ab04316ba77a467b4 P c9b7994f388844d19db8c7741249d3c2: Deleting consensus metadata
I20260128 20:51:01.094631 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9736
I20260128 20:51:01.096571 9524 catalog_manager.cc:4985] TS c9b7994f388844d19db8c7741249d3c2 (127.4.183.131:38701): tablet 39ac60c98be0420ab04316ba77a467b4 (table test-table [id=3fe87f271bfd4a108ca09835bb6bec05]) successfully deleted
I20260128 20:51:01.097259 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.096888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:36317 (local address 127.0.0.1:34532)
0128 20:51:01.097031 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:01.097105 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.129:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260128 20:51:01.097420 9522 catalog_manager.cc:4712] TS 60a097b8e2c547a08a568c9a893969d8 (127.4.183.129:36317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39ac60c98be0420ab04316ba77a467b4: Network error: Client connection negotiation failed: client connection to 127.4.183.129:36317: connect: Connection refused (error 111)
I20260128 20:51:01.101065 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9871
I20260128 20:51:01.107715 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 9506
2026-01-28T20:51:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3501 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-01-28T20:51:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:01Z Disabled control of system clock
I20260128 20:51:01.142304 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:43617
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:43617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260128 20:51:01.267158 10047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:01.267583 10047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:01.267685 10047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:01.271548 10047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:01.271692 10047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:01.271723 10047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:01.271763 10047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:01.271823 10047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:01.276420 10047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:43617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:01.277756 10047 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:01.278996 10047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:01.285677 10053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:01.285671 10055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:01.285895 10047 server_base.cc:1060] running on GCE node
W20260128 20:51:01.285671 10052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:01.286518 10047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:01.287657 10047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:01.288868 10047 hybrid_clock.cc:648] HybridClock initialized: now 1769633461288845 us; error 48 us; skew 500 ppm
I20260128 20:51:01.291026 10047 webserver.cc:492] Webserver started at http://127.4.183.190:37961/ using document root <none> and password file <none>
I20260128 20:51:01.291694 10047 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:01.291790 10047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:01.292065 10047 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:01.294088 10047 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "82544d52fe5f44a7a2244ac4bb653be9"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "d22f87139a5953ff54eeed49695024b1"
server_key_iv: "021432aec252465f105ba3e0c9770c14"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.294674 10047 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "82544d52fe5f44a7a2244ac4bb653be9"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "d22f87139a5953ff54eeed49695024b1"
server_key_iv: "021432aec252465f105ba3e0c9770c14"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.298733 10047 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:01.301821 10061 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.303150 10047 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:01.303342 10047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "82544d52fe5f44a7a2244ac4bb653be9"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "d22f87139a5953ff54eeed49695024b1"
server_key_iv: "021432aec252465f105ba3e0c9770c14"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.303473 10047 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:01.308915 10047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:01.309676 10047 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:01.309881 10047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:01.318182 10047 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:43617
I20260128 20:51:01.318225 10113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:43617 every 8 connection(s)
I20260128 20:51:01.319303 10047 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:01.320559 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10047
I20260128 20:51:01.320712 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:01.320971 4830 external_mini_cluster.cc:1468] Setting key f805ad39b07379d57ec4c763437a0e9b
I20260128 20:51:01.323016 10114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:01.327248 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.322387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.4.183.190:43617)
0128 20:51:01.323564 (+ 1177us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.323575 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.323630 (+ 55us) server_negotiation.cc:408] Connection header received
0128 20:51:01.324290 (+ 660us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.324308 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.324672 (+ 364us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.325059 (+ 387us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:01.326025 (+ 966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:01.326030 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:01.326040 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:01.326052 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:01.326097 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:01.326109 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:01.326282 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.326383 (+ 101us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.326641 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":147,"threads_started":1}
I20260128 20:51:01.329833 10114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: Bootstrap starting.
I20260128 20:51:01.332221 10114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:01.333094 10114 log.cc:826] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:01.336067 10114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: No bootstrap required, opened a new log
I20260128 20:51:01.340137 10114 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82544d52fe5f44a7a2244ac4bb653be9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43617 } }
I20260128 20:51:01.340461 10114 raft_consensus.cc:385] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:01.340502 10114 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82544d52fe5f44a7a2244ac4bb653be9, State: Initialized, Role: FOLLOWER
I20260128 20:51:01.340998 10114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82544d52fe5f44a7a2244ac4bb653be9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43617 } }
I20260128 20:51:01.341183 10114 raft_consensus.cc:399] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:01.341270 10114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:01.341403 10114 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:01.342468 10114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82544d52fe5f44a7a2244ac4bb653be9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43617 } }
I20260128 20:51:01.342898 10114 leader_election.cc:304] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82544d52fe5f44a7a2244ac4bb653be9; no voters:
I20260128 20:51:01.343255 10114 leader_election.cc:290] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:01.343483 10119 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:01.343729 10119 raft_consensus.cc:697] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [term 1 LEADER]: Becoming Leader. State: Replica: 82544d52fe5f44a7a2244ac4bb653be9, State: Running, Role: LEADER
I20260128 20:51:01.344069 10119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82544d52fe5f44a7a2244ac4bb653be9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43617 } }
I20260128 20:51:01.344642 10114 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:01.346222 10121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82544d52fe5f44a7a2244ac4bb653be9. Latest consensus state: current_term: 1 leader_uuid: "82544d52fe5f44a7a2244ac4bb653be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82544d52fe5f44a7a2244ac4bb653be9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43617 } } }
I20260128 20:51:01.346253 10120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82544d52fe5f44a7a2244ac4bb653be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82544d52fe5f44a7a2244ac4bb653be9" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 43617 } } }
I20260128 20:51:01.346374 10120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:01.346375 10121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9 [sys.catalog]: This master's current role is: LEADER
W20260128 20:51:01.349946 10134 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:01.350055 10134 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:01.350176 10131 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:01.351164 10131 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:01.357445 10131 catalog_manager.cc:1357] Generated new cluster ID: fd77e9c6d7e6410699a61cf25d11b469
I20260128 20:51:01.357548 10131 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:01.372838 10131 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:01.373271 10131 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:01.386462 10131 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: Generated new TSK 0
I20260128 20:51:01.387475 10131 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:51:01.396900 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43617
--builtin_ntp_servers=127.4.183.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260128 20:51:01.516578 10138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:01.516856 10138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:01.516916 10138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:01.520596 10138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:01.520700 10138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:01.520783 10138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:01.525178 10138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:01.526513 10138 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:01.527853 10138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:01.535202 10143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:01.535210 10146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:01.535210 10144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:01.535966 10138 server_base.cc:1060] running on GCE node
I20260128 20:51:01.536486 10138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:01.537341 10138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:01.538571 10138 hybrid_clock.cc:648] HybridClock initialized: now 1769633461538539 us; error 53 us; skew 500 ppm
I20260128 20:51:01.541482 10138 webserver.cc:492] Webserver started at http://127.4.183.129:43141/ using document root <none> and password file <none>
I20260128 20:51:01.542299 10138 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:01.542424 10138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:01.542680 10138 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:01.545405 10138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "f6e61e95df7e4683ae90ba32072129a9"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "f7cb810bbdbe8440794e8de9ced7f0d4"
server_key_iv: "34e8d2844d8cd029382c0df7d1e2e4db"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.546077 10138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "f6e61e95df7e4683ae90ba32072129a9"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "f7cb810bbdbe8440794e8de9ced7f0d4"
server_key_iv: "34e8d2844d8cd029382c0df7d1e2e4db"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.550845 10138 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:01.554884 10152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.556303 10138 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260128 20:51:01.556489 10138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "f6e61e95df7e4683ae90ba32072129a9"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "f7cb810bbdbe8440794e8de9ced7f0d4"
server_key_iv: "34e8d2844d8cd029382c0df7d1e2e4db"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.556668 10138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:01.562326 10138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:01.563051 10138 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:01.563263 10138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:01.563896 10138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:01.564983 10138 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:01.565033 10138 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.565075 10138 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:01.565155 10138 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260128 20:51:01.577116 10138 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:38039
I20260128 20:51:01.577680 10265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:38039 every 8 connection(s)
I20260128 20:51:01.578907 10138 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:01.584256 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10138
I20260128 20:51:01.584389 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:01.584673 4830 external_mini_cluster.cc:1468] Setting key dde1ab219794ae6a5364a7c3e4fddafe
I20260128 20:51:01.586014 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.581569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:59689 (local address 127.4.183.190:43617)
0128 20:51:01.581802 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.581806 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.582818 (+ 1012us) server_negotiation.cc:408] Connection header received
0128 20:51:01.583739 (+ 921us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.583743 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.583793 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.583875 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:01.585436 (+ 1561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:01.585440 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:01.585442 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:01.585458 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:01.585485 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:01.585495 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:01.585593 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.585844 (+ 251us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.585890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260128 20:51:01.586966 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.581817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43617 (local address 127.4.183.129:59689)
0128 20:51:01.582629 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:01.582678 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:01.583497 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:01.584075 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:01.584084 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:01.584184 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:01.584945 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:01.584947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:01.585001 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:01.585082 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:01.585618 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:01.585627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:01.585635 (+ 8us) client_negotiation.cc:769] Sending connection context
0128 20:51:01.585897 (+ 262us) client_negotiation.cc:240] Negotiation successful
0128 20:51:01.586161 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":149,"threads_started":1}
I20260128 20:51:01.587742 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43617
--builtin_ntp_servers=127.4.183.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260128 20:51:01.588438 10266 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43617
I20260128 20:51:01.588891 10266 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:01.589679 10266 heartbeater.cc:507] Master 127.4.183.190:43617 requested a full tablet report, sending...
I20260128 20:51:01.591758 10078 ts_manager.cc:194] Registered new tserver with Master: f6e61e95df7e4683ae90ba32072129a9 (127.4.183.129:38039)
W20260128 20:51:01.716244 10269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:01.716674 10269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:01.716809 10269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:01.721478 10269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:01.721602 10269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:01.721730 10269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:01.726191 10269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:01.727530 10269 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:01.728821 10269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:01.736985 10277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:01.736985 10275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:01.737062 10269 server_base.cc:1060] running on GCE node
W20260128 20:51:01.736992 10274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:01.737812 10269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:01.738502 10269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:01.739719 10269 hybrid_clock.cc:648] HybridClock initialized: now 1769633461739690 us; error 51 us; skew 500 ppm
I20260128 20:51:01.742064 10269 webserver.cc:492] Webserver started at http://127.4.183.130:41863/ using document root <none> and password file <none>
I20260128 20:51:01.743206 10269 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:01.743377 10269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:01.743685 10269 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:01.745975 10269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "252d067d43eb41ecab403f6965192862"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "2a2d6a617f787d3b4747f1fea334c896"
server_key_iv: "7abfeee22fbf1242769903cfa253a992"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.746577 10269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "252d067d43eb41ecab403f6965192862"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "2a2d6a617f787d3b4747f1fea334c896"
server_key_iv: "7abfeee22fbf1242769903cfa253a992"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.750692 10269 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:01.753563 10283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.754772 10269 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:51:01.754942 10269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "252d067d43eb41ecab403f6965192862"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "2a2d6a617f787d3b4747f1fea334c896"
server_key_iv: "7abfeee22fbf1242769903cfa253a992"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.755069 10269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:01.761370 10269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:01.762135 10269 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:01.762355 10269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:01.762976 10269 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:01.764111 10269 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:01.764196 10269 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.764276 10269 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:01.764325 10269 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.775285 10269 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36369
I20260128 20:51:01.775305 10396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36369 every 8 connection(s)
I20260128 20:51:01.776497 10269 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:01.786528 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10269
I20260128 20:51:01.786701 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:01.787002 4830 external_mini_cluster.cc:1468] Setting key 0007404b555257116d6ddbd4891ee2bc
I20260128 20:51:01.787338 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.778680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:43117 (local address 127.4.183.190:43617)
0128 20:51:01.778861 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.778866 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.780059 (+ 1193us) server_negotiation.cc:408] Connection header received
0128 20:51:01.780903 (+ 844us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.780907 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.780950 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.781034 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:01.786696 (+ 5662us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:01.786702 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:01.786706 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:01.786731 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:01.786763 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:01.786776 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:01.786903 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.787124 (+ 221us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.787186 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260128 20:51:01.788412 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.779065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43617 (local address 127.4.183.130:43117)
0128 20:51:01.779873 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:01.779916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:01.780703 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:01.781236 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:01.781245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:01.781366 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:01.786256 (+ 4890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:01.786259 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:01.786317 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:01.786428 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:01.786914 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:01.786931 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:01.786936 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 20:51:01.787206 (+ 270us) client_negotiation.cc:240] Negotiation successful
0128 20:51:01.787520 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":146,"threads_started":1}
I20260128 20:51:01.789183 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43617
--builtin_ntp_servers=127.4.183.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260128 20:51:01.789784 10397 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43617
I20260128 20:51:01.790087 10397 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:01.790657 10397 heartbeater.cc:507] Master 127.4.183.190:43617 requested a full tablet report, sending...
I20260128 20:51:01.792091 10078 ts_manager.cc:194] Registered new tserver with Master: 252d067d43eb41ecab403f6965192862 (127.4.183.130:36369)
W20260128 20:51:01.912006 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:01.912314 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:01.912375 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:01.916323 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:01.916409 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:01.916528 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:01.920817 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:01.922153 10400 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:01.923416 10400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:01.931166 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:01.931173 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:01.931167 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:01.931283 10400 server_base.cc:1060] running on GCE node
I20260128 20:51:01.931851 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:01.932541 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:01.933732 10400 hybrid_clock.cc:648] HybridClock initialized: now 1769633461933703 us; error 51 us; skew 500 ppm
I20260128 20:51:01.935866 10400 webserver.cc:492] Webserver started at http://127.4.183.131:41953/ using document root <none> and password file <none>
I20260128 20:51:01.936506 10400 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:01.936594 10400 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:01.936834 10400 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:01.938724 10400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "8977d543270f4bf79158d62394bbf4d4"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "a42f60f168b583406ff8d29aacc1e4ef"
server_key_iv: "2aad5bf872ff67347d2d171fa6fdbc90"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.939260 10400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "8977d543270f4bf79158d62394bbf4d4"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "a42f60f168b583406ff8d29aacc1e4ef"
server_key_iv: "2aad5bf872ff67347d2d171fa6fdbc90"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.943702 10400 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:51:01.946691 10414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.947947 10400 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:01.948122 10400 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "8977d543270f4bf79158d62394bbf4d4"
format_stamp: "Formatted at 2026-01-28 20:51:01 on dist-test-slave-hss7"
server_key: "a42f60f168b583406ff8d29aacc1e4ef"
server_key_iv: "2aad5bf872ff67347d2d171fa6fdbc90"
server_key_version: "encryptionkey@0"
I20260128 20:51:01.948252 10400 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:01.953632 10400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:01.954315 10400 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:01.954524 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:01.955093 10400 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:01.956108 10400 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:01.956182 10400 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.956254 10400 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:01.956305 10400 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:01.967619 10400 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:33511
I20260128 20:51:01.967684 10527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:33511 every 8 connection(s)
I20260128 20:51:01.968739 10400 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:01.976107 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.971332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:51455 (local address 127.4.183.190:43617)
0128 20:51:01.971540 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.971545 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.972841 (+ 1296us) server_negotiation.cc:408] Connection header received
0128 20:51:01.973910 (+ 1069us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.973914 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.973966 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.974056 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:01.975548 (+ 1492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:01.975551 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:01.975553 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:01.975567 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:01.975589 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:01.975597 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:01.975728 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.975898 (+ 170us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.975953 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260128 20:51:01.977079 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.971755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:43617 (local address 127.4.183.131:51455)
0128 20:51:01.972644 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:01.972691 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:01.973645 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:01.974262 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:01.974272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:01.974404 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:01.975229 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:01.975231 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:01.975273 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:01.975341 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:01.975711 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:01.975719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:01.975724 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 20:51:01.975973 (+ 249us) client_negotiation.cc:240] Negotiation successful
0128 20:51:01.976225 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":136,"threads_started":1}
I20260128 20:51:01.977223 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10400
I20260128 20:51:01.977340 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:01.977660 4830 external_mini_cluster.cc:1468] Setting key 8e054adb429fa96a45d2f8b086ebcec5
I20260128 20:51:01.978529 10528 heartbeater.cc:344] Connected to a master server at 127.4.183.190:43617
I20260128 20:51:01.978884 10528 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:01.979815 10528 heartbeater.cc:507] Master 127.4.183.190:43617 requested a full tablet report, sending...
I20260128 20:51:01.982271 10078 ts_manager.cc:194] Registered new tserver with Master: 8977d543270f4bf79158d62394bbf4d4 (127.4.183.131:33511)
I20260128 20:51:01.993088 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 20:51:01.998162 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:01.995572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.4.183.190:43617)
0128 20:51:01.995864 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:01.995868 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:01.995989 (+ 121us) server_negotiation.cc:408] Connection header received
0128 20:51:01.996101 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:01.996103 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:01.996144 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:01.996227 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:01.997518 (+ 1291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:01.997522 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:01.997525 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:01.997545 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:01.997584 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:01.997593 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:01.997709 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:01.997906 (+ 197us) server_negotiation.cc:300] Negotiation successful
0128 20:51:01.997950 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260128 20:51:02.001915 10078 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:02.004674 10078 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:02.019243 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.015382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:38039 (local address 127.0.0.1:47566)
0128 20:51:02.016319 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.016377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:02.016596 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:02.017408 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:02.017411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:02.017528 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:02.018326 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:02.018328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:02.018337 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:02.018360 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:02.018849 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:02.018857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:02.018861 (+ 4us) client_negotiation.cc:769] Sending connection context
0128 20:51:02.019001 (+ 140us) client_negotiation.cc:240] Negotiation successful
0128 20:51:02.019077 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":122,"threads_started":1}
I20260128 20:51:02.019258 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.015513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.4.183.129:38039)
0128 20:51:02.016452 (+ 939us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.016460 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.016492 (+ 32us) server_negotiation.cc:408] Connection header received
0128 20:51:02.016798 (+ 306us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.016806 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.017019 (+ 213us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.017237 (+ 218us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.018550 (+ 1313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.018557 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.018569 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.018582 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.018649 (+ 67us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.018665 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.018798 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.019034 (+ 236us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.019112 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":96,"threads_started":1}
I20260128 20:51:02.019948 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.015675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.4.183.130:36369)
0128 20:51:02.016711 (+ 1036us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.016719 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.016744 (+ 25us) server_negotiation.cc:408] Connection header received
0128 20:51:02.016806 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.016812 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.017017 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.017793 (+ 776us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.018625 (+ 832us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.018634 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.018652 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.018673 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.018752 (+ 79us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.018769 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.019221 (+ 452us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.019671 (+ 450us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.019748 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260128 20:51:02.019564 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.015465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:36369 (local address 127.0.0.1:59586)
0128 20:51:02.016364 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.016381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:02.016498 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:02.017278 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:02.017285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:02.017363 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:02.018178 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:02.018180 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:02.018221 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:02.018291 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:02.019275 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:02.019280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:02.019286 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 20:51:02.019369 (+ 83us) client_negotiation.cc:240] Negotiation successful
0128 20:51:02.019417 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":153,"threads_started":1}
I20260128 20:51:02.020506 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.016783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.4.183.131:33511)
0128 20:51:02.017278 (+ 495us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.017286 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.017553 (+ 267us) server_negotiation.cc:408] Connection header received
0128 20:51:02.017752 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.017759 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.017949 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.018069 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.019659 (+ 1590us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.019667 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.019679 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.019694 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.019750 (+ 56us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.019764 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.019888 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.020296 (+ 408us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.020368 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":156,"threads_started":1}
I20260128 20:51:02.020955 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.016200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:33511 (local address 127.0.0.1:49462)
0128 20:51:02.017451 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.017469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:02.017617 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:02.018576 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:02.018580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:02.018594 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:02.019463 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:02.019465 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:02.019478 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:02.019503 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:02.019877 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:02.019880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:02.019884 (+ 4us) client_negotiation.cc:769] Sending connection context
0128 20:51:02.019976 (+ 92us) client_negotiation.cc:240] Negotiation successful
0128 20:51:02.020822 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":100,"threads_started":1}
I20260128 20:51:02.023453 10200 tablet_service.cc:1505] Processing CreateTablet for tablet 16120f955a0946a19bbbb878f08da2e4 (DEFAULT_TABLE table=test-table [id=88bc4afd32034a209324e8a408763242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:02.023452 10331 tablet_service.cc:1505] Processing CreateTablet for tablet 16120f955a0946a19bbbb878f08da2e4 (DEFAULT_TABLE table=test-table [id=88bc4afd32034a209324e8a408763242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:02.023634 10462 tablet_service.cc:1505] Processing CreateTablet for tablet 16120f955a0946a19bbbb878f08da2e4 (DEFAULT_TABLE table=test-table [id=88bc4afd32034a209324e8a408763242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:02.024653 10462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16120f955a0946a19bbbb878f08da2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:02.024653 10200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16120f955a0946a19bbbb878f08da2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:02.024653 10331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16120f955a0946a19bbbb878f08da2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:02.031809 10545 tablet_bootstrap.cc:492] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Bootstrap starting.
I20260128 20:51:02.032897 10546 tablet_bootstrap.cc:492] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Bootstrap starting.
I20260128 20:51:02.032898 10547 tablet_bootstrap.cc:492] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: Bootstrap starting.
I20260128 20:51:02.035599 10545 tablet_bootstrap.cc:654] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:02.035604 10547 tablet_bootstrap.cc:654] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:02.035604 10546 tablet_bootstrap.cc:654] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:02.036564 10545 log.cc:826] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:02.036594 10546 log.cc:826] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:02.036590 10547 log.cc:826] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:02.038831 10545 tablet_bootstrap.cc:492] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: No bootstrap required, opened a new log
I20260128 20:51:02.038831 10546 tablet_bootstrap.cc:492] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: No bootstrap required, opened a new log
I20260128 20:51:02.039125 10546 ts_tablet_manager.cc:1403] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:51:02.039126 10545 ts_tablet_manager.cc:1403] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260128 20:51:02.041090 10547 tablet_bootstrap.cc:492] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: No bootstrap required, opened a new log
I20260128 20:51:02.041390 10547 ts_tablet_manager.cc:1403] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260128 20:51:02.042423 10546 raft_consensus.cc:359] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.042579 10545 raft_consensus.cc:359] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.042743 10546 raft_consensus.cc:385] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:02.042805 10546 raft_consensus.cc:740] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252d067d43eb41ecab403f6965192862, State: Initialized, Role: FOLLOWER
I20260128 20:51:02.042814 10545 raft_consensus.cc:385] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:02.042861 10545 raft_consensus.cc:740] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8977d543270f4bf79158d62394bbf4d4, State: Initialized, Role: FOLLOWER
I20260128 20:51:02.043309 10546 consensus_queue.cc:260] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.043326 10545 consensus_queue.cc:260] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.044337 10528 heartbeater.cc:499] Master 127.4.183.190:43617 was elected leader, sending a full tablet report...
I20260128 20:51:02.044400 10546 ts_tablet_manager.cc:1434] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260128 20:51:02.044400 10545 ts_tablet_manager.cc:1434] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260128 20:51:02.044658 10397 heartbeater.cc:499] Master 127.4.183.190:43617 was elected leader, sending a full tablet report...
I20260128 20:51:02.044899 10547 raft_consensus.cc:359] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.045148 10547 raft_consensus.cc:385] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:02.045226 10547 raft_consensus.cc:740] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e61e95df7e4683ae90ba32072129a9, State: Initialized, Role: FOLLOWER
I20260128 20:51:02.045800 10547 consensus_queue.cc:260] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.046933 10547 ts_tablet_manager.cc:1434] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260128 20:51:02.046958 10266 heartbeater.cc:499] Master 127.4.183.190:43617 was elected leader, sending a full tablet report...
I20260128 20:51:02.055518 10551 raft_consensus.cc:493] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:02.055795 10551 raft_consensus.cc:515] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.058046 10551 leader_election.cc:290] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6e61e95df7e4683ae90ba32072129a9 (127.4.183.129:38039), 8977d543270f4bf79158d62394bbf4d4 (127.4.183.131:33511)
I20260128 20:51:02.060541 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.058445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:54741 (local address 127.4.183.129:38039)
0128 20:51:02.058603 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.058610 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.058632 (+ 22us) server_negotiation.cc:408] Connection header received
0128 20:51:02.058696 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.058700 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.058747 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.058855 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.059892 (+ 1037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.059897 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.059909 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.059924 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.059952 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.059966 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.060069 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.060284 (+ 215us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.060414 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260128 20:51:02.060565 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.058189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:38039 (local address 127.4.183.130:54741)
0128 20:51:02.058363 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.058391 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:02.058488 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:02.059014 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:02.059017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:02.059088 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:02.059671 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:02.059673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:02.059688 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:02.059717 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:02.060092 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:02.060097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:02.060104 (+ 7us) client_negotiation.cc:769] Sending connection context
0128 20:51:02.060222 (+ 118us) client_negotiation.cc:240] Negotiation successful
0128 20:51:02.060270 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260128 20:51:02.061499 10220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16120f955a0946a19bbbb878f08da2e4" candidate_uuid: "252d067d43eb41ecab403f6965192862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e61e95df7e4683ae90ba32072129a9" is_pre_election: true
I20260128 20:51:02.061743 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.058569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:58553 (local address 127.4.183.131:33511)
0128 20:51:02.058749 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.058753 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.059296 (+ 543us) server_negotiation.cc:408] Connection header received
0128 20:51:02.059398 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.059402 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.059465 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.059563 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.061091 (+ 1528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.061096 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.061103 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.061124 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.061214 (+ 90us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.061227 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.061345 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.061522 (+ 177us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.061574 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 20:51:02.061775 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.058430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:33511 (local address 127.4.183.130:58553)
0128 20:51:02.059150 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.059168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:02.059278 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:02.059624 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:02.059630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:02.059659 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:02.060815 (+ 1156us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:02.060818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:02.060833 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:02.060863 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:02.061400 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:02.061404 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:02.061407 (+ 3us) client_negotiation.cc:769] Sending connection context
0128 20:51:02.061523 (+ 116us) client_negotiation.cc:240] Negotiation successful
0128 20:51:02.061590 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":222,"threads_started":1}
I20260128 20:51:02.061896 10220 raft_consensus.cc:2468] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252d067d43eb41ecab403f6965192862 in term 0.
I20260128 20:51:02.062556 10286 leader_election.cc:304] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252d067d43eb41ecab403f6965192862, f6e61e95df7e4683ae90ba32072129a9; no voters:
I20260128 20:51:02.062577 10482 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16120f955a0946a19bbbb878f08da2e4" candidate_uuid: "252d067d43eb41ecab403f6965192862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8977d543270f4bf79158d62394bbf4d4" is_pre_election: true
I20260128 20:51:02.062880 10482 raft_consensus.cc:2468] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252d067d43eb41ecab403f6965192862 in term 0.
I20260128 20:51:02.063258 10551 raft_consensus.cc:2804] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:02.063359 10551 raft_consensus.cc:493] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:02.063397 10551 raft_consensus.cc:3060] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:02.064463 10551 raft_consensus.cc:515] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.064949 10551 leader_election.cc:290] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [CANDIDATE]: Term 1 election: Requested vote from peers f6e61e95df7e4683ae90ba32072129a9 (127.4.183.129:38039), 8977d543270f4bf79158d62394bbf4d4 (127.4.183.131:33511)
I20260128 20:51:02.065642 10220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16120f955a0946a19bbbb878f08da2e4" candidate_uuid: "252d067d43eb41ecab403f6965192862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e61e95df7e4683ae90ba32072129a9"
I20260128 20:51:02.065642 10482 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16120f955a0946a19bbbb878f08da2e4" candidate_uuid: "252d067d43eb41ecab403f6965192862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8977d543270f4bf79158d62394bbf4d4"
I20260128 20:51:02.065804 10220 raft_consensus.cc:3060] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:02.065804 10482 raft_consensus.cc:3060] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:02.067791 10220 raft_consensus.cc:2468] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252d067d43eb41ecab403f6965192862 in term 1.
I20260128 20:51:02.067791 10482 raft_consensus.cc:2468] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252d067d43eb41ecab403f6965192862 in term 1.
I20260128 20:51:02.068341 10286 leader_election.cc:304] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 252d067d43eb41ecab403f6965192862, 8977d543270f4bf79158d62394bbf4d4, f6e61e95df7e4683ae90ba32072129a9; no voters:
I20260128 20:51:02.068578 10551 raft_consensus.cc:2804] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:02.068871 10551 raft_consensus.cc:697] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 1 LEADER]: Becoming Leader. State: Replica: 252d067d43eb41ecab403f6965192862, State: Running, Role: LEADER
I20260128 20:51:02.069350 10551 consensus_queue.cc:237] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } }
I20260128 20:51:02.073297 10078 catalog_manager.cc:5654] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 reported cstate change: term changed from 0 to 1, leader changed from <none> to 252d067d43eb41ecab403f6965192862 (127.4.183.130). New cstate: current_term: 1 leader_uuid: "252d067d43eb41ecab403f6965192862" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252d067d43eb41ecab403f6965192862" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 36369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 } health_report { overall_health: UNKNOWN } } }
W20260128 20:51:02.080688 10267 tablet.cc:2378] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:02.089690 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.087909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.4.183.130:36369)
0128 20:51:02.088062 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.088067 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.088103 (+ 36us) server_negotiation.cc:408] Connection header received
0128 20:51:02.088205 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.088207 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.088268 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.088364 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.089123 (+ 759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.089127 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.089178 (+ 51us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.089194 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.089219 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.089236 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.089343 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.089473 (+ 130us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.089518 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 20:51:02.096338 10220 raft_consensus.cc:1275] T 16120f955a0946a19bbbb878f08da2e4 P f6e61e95df7e4683ae90ba32072129a9 [term 1 FOLLOWER]: Refusing update from remote peer 252d067d43eb41ecab403f6965192862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:02.096354 10482 raft_consensus.cc:1275] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 1 FOLLOWER]: Refusing update from remote peer 252d067d43eb41ecab403f6965192862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:02.097203 10551 consensus_queue.cc:1048] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8977d543270f4bf79158d62394bbf4d4" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 33511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:02.097477 10555 consensus_queue.cc:1048] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e61e95df7e4683ae90ba32072129a9" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:02.106173 10559 mvcc.cc:204] Tried to move back new op lower bound from 7248418660737687552 to 7248418660639571968. Current Snapshot: MvccSnapshot[applied={T|T < 7248418660737687552}]
I20260128 20:51:02.115746 10077 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:02.115981 10077 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:02.119086 10077 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 82544d52fe5f44a7a2244ac4bb653be9: Sending DeleteTablet for 3 replicas of tablet 16120f955a0946a19bbbb878f08da2e4
I20260128 20:51:02.120165 10331 tablet_service.cc:1552] Processing DeleteTablet for tablet 16120f955a0946a19bbbb878f08da2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:02 UTC) from {username='slave'} at 127.0.0.1:59586
I20260128 20:51:02.120183 10200 tablet_service.cc:1552] Processing DeleteTablet for tablet 16120f955a0946a19bbbb878f08da2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:02 UTC) from {username='slave'} at 127.0.0.1:47566
I20260128 20:51:02.120601 10462 tablet_service.cc:1552] Processing DeleteTablet for tablet 16120f955a0946a19bbbb878f08da2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:02 UTC) from {username='slave'} at 127.0.0.1:49462
I20260128 20:51:02.120925 10568 tablet_replica.cc:333] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: stopping tablet replica
I20260128 20:51:02.121246 10570 tablet_replica.cc:333] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: stopping tablet replica
I20260128 20:51:02.121467 10570 raft_consensus.cc:2243] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:02.121461 10568 raft_consensus.cc:2243] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:02.121793 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10138
I20260128 20:51:02.122483 10570 raft_consensus.cc:2272] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:02.122915 10568 raft_consensus.cc:2272] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:02.124542 10568 ts_tablet_manager.cc:1916] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:02.124542 10570 ts_tablet_manager.cc:1916] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:02.127494 10568 ts_tablet_manager.cc:1929] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:02.127494 10570 ts_tablet_manager.cc:1929] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:02.127629 10568 log.cc:1199] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/16120f955a0946a19bbbb878f08da2e4
I20260128 20:51:02.127629 10570 log.cc:1199] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/16120f955a0946a19bbbb878f08da2e4
I20260128 20:51:02.127977 10570 ts_tablet_manager.cc:1950] T 16120f955a0946a19bbbb878f08da2e4 P 8977d543270f4bf79158d62394bbf4d4: Deleting consensus metadata
I20260128 20:51:02.127977 10568 ts_tablet_manager.cc:1950] T 16120f955a0946a19bbbb878f08da2e4 P 252d067d43eb41ecab403f6965192862: Deleting consensus metadata
I20260128 20:51:02.129383 10065 catalog_manager.cc:4985] TS 252d067d43eb41ecab403f6965192862 (127.4.183.130:36369): tablet 16120f955a0946a19bbbb878f08da2e4 (table test-table [id=88bc4afd32034a209324e8a408763242]) successfully deleted
I20260128 20:51:02.129383 10063 catalog_manager.cc:4985] TS 8977d543270f4bf79158d62394bbf4d4 (127.4.183.131:33511): tablet 16120f955a0946a19bbbb878f08da2e4 (table test-table [id=88bc4afd32034a209324e8a408763242]) successfully deleted
W20260128 20:51:02.132230 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.183.129:38039 (error 108)
I20260128 20:51:02.133381 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10269
I20260128 20:51:02.133932 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.133411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:38039 (local address 127.0.0.1:47582)
0128 20:51:02.133645 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.133769 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.129:38039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260128 20:51:02.134603 10064 catalog_manager.cc:4712] TS f6e61e95df7e4683ae90ba32072129a9 (127.4.183.129:38039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16120f955a0946a19bbbb878f08da2e4: Network error: Client connection negotiation failed: client connection to 127.4.183.129:38039: connect: Connection refused (error 111)
I20260128 20:51:02.142719 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10400
I20260128 20:51:02.149049 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10047
2026-01-28T20:51:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1040 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260128 20:51:02.172497 4830 mini_oidc.cc:150] Starting JWKS server
I20260128 20:51:02.174198 4830 webserver.cc:492] Webserver started at https://127.0.0.1:36679/ using document root <none> and password file <none>
I20260128 20:51:02.174392 4830 mini_oidc.cc:186] Starting OIDC Discovery server
I20260128 20:51:02.174718 4830 webserver.cc:492] Webserver started at http://127.0.0.1:36309/ using document root <none> and password file <none>
2026-01-28T20:51:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:02Z Disabled control of system clock
I20260128 20:51:02.184288 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:41373
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:35795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:41373
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/testchainca.cert with env {}
W20260128 20:51:02.306931 10577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:02.307211 10577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:02.307265 10577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:02.311043 10577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:02.311132 10577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:02.311149 10577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:02.311169 10577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:02.311187 10577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 20:51:02.311205 10577 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 20:51:02.311220 10577 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36679/jwks.json
I20260128 20:51:02.316270 10577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:35795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:41373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:02.317595 10577 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:02.318480 10577 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:02.325139 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:02.325234 10585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:02.325155 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:02.325487 10577 server_base.cc:1060] running on GCE node
I20260128 20:51:02.326045 10577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:02.327121 10577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:02.328406 10577 hybrid_clock.cc:648] HybridClock initialized: now 1769633462328386 us; error 57 us; skew 500 ppm
I20260128 20:51:02.331255 10577 webserver.cc:492] Webserver started at http://127.4.183.190:40579/ using document root <none> and password file <none>
I20260128 20:51:02.331923 10577 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:02.331995 10577 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:02.332180 10577 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:02.334245 10577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "dd3220c791d84b7e95d27ccba416838c"
format_stamp: "Formatted at 2026-01-28 20:51:02 on dist-test-slave-hss7"
server_key: "3e71d26eb3d845279d68efa8df90fc52"
server_key_iv: "d713518a9b4898b12e95fa51189ef8ed"
server_key_version: "encryptionkey@0"
I20260128 20:51:02.334801 10577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "dd3220c791d84b7e95d27ccba416838c"
format_stamp: "Formatted at 2026-01-28 20:51:02 on dist-test-slave-hss7"
server_key: "3e71d26eb3d845279d68efa8df90fc52"
server_key_iv: "d713518a9b4898b12e95fa51189ef8ed"
server_key_version: "encryptionkey@0"
I20260128 20:51:02.338769 10577 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:02.341637 10591 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:02.342986 10577 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:51:02.343139 10577 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "dd3220c791d84b7e95d27ccba416838c"
format_stamp: "Formatted at 2026-01-28 20:51:02 on dist-test-slave-hss7"
server_key: "3e71d26eb3d845279d68efa8df90fc52"
server_key_iv: "d713518a9b4898b12e95fa51189ef8ed"
server_key_version: "encryptionkey@0"
I20260128 20:51:02.343250 10577 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:02.361332 10577 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:02.362331 10577 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:02.362540 10577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:02.372205 10577 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:41373
I20260128 20:51:02.372292 10643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:41373 every 8 connection(s)
I20260128 20:51:02.373478 10577 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:02.377285 10644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:02.381533 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10577
I20260128 20:51:02.381709 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:02.382009 4830 external_mini_cluster.cc:1468] Setting key 145bf84499f26f0db742c582f5bad678
I20260128 20:51:02.385864 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c: Bootstrap starting.
I20260128 20:51:02.388921 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:02.390087 10644 log.cc:826] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:02.392609 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c: No bootstrap required, opened a new log
I20260128 20:51:02.392620 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.384052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.4.183.190:41373)
0128 20:51:02.385323 (+ 1271us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.385334 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.385386 (+ 52us) server_negotiation.cc:408] Connection header received
0128 20:51:02.386087 (+ 701us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.386120 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.386571 (+ 451us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.386938 (+ 367us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:02.387745 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.388579 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:02.389674 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.389986 (+ 312us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:02.390711 (+ 725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:02.390735 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:02.390749 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:02.391181 (+ 432us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:02.391235 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:02.391250 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:02.391355 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.391704 (+ 349us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.391974 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":263,"threads_started":1}
I20260128 20:51:02.396111 10644 raft_consensus.cc:359] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3220c791d84b7e95d27ccba416838c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 41373 } }
I20260128 20:51:02.396402 10644 raft_consensus.cc:385] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:02.396492 10644 raft_consensus.cc:740] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3220c791d84b7e95d27ccba416838c, State: Initialized, Role: FOLLOWER
I20260128 20:51:02.396775 4830 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 20:51:02.397045 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3220c791d84b7e95d27ccba416838c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 41373 } }
I20260128 20:51:02.397301 10644 raft_consensus.cc:399] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:02.397385 10644 raft_consensus.cc:493] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:02.397507 10644 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:02.398694 10644 raft_consensus.cc:515] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3220c791d84b7e95d27ccba416838c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 41373 } }
I20260128 20:51:02.399153 10644 leader_election.cc:304] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd3220c791d84b7e95d27ccba416838c; no voters:
I20260128 20:51:02.399513 10644 leader_election.cc:290] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:02.399701 10649 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:02.399957 10649 raft_consensus.cc:697] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [term 1 LEADER]: Becoming Leader. State: Replica: dd3220c791d84b7e95d27ccba416838c, State: Running, Role: LEADER
I20260128 20:51:02.400280 10649 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3220c791d84b7e95d27ccba416838c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 41373 } }
I20260128 20:51:02.400826 10644 sys_catalog.cc:565] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:02.402118 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd3220c791d84b7e95d27ccba416838c. Latest consensus state: current_term: 1 leader_uuid: "dd3220c791d84b7e95d27ccba416838c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3220c791d84b7e95d27ccba416838c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 41373 } } }
I20260128 20:51:02.402266 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:02.402343 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd3220c791d84b7e95d27ccba416838c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3220c791d84b7e95d27ccba416838c" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 41373 } } }
I20260128 20:51:02.402446 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:02.403775 10658 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:02.407192 10658 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:02.413885 10658 catalog_manager.cc:1357] Generated new cluster ID: 07d0cf4f75a9493e9d0fb397eb0d0335
I20260128 20:51:02.414041 10658 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:02.434056 10658 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:02.435520 10658 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:02.447182 10658 catalog_manager.cc:6027] T 00000000000000000000000000000000 P dd3220c791d84b7e95d27ccba416838c: Generated new TSK 0
I20260128 20:51:02.448220 10658 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:51:02.490041 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.466281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.4.183.190:41373)
0128 20:51:02.466494 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.466498 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.466689 (+ 191us) server_negotiation.cc:408] Connection header received
0128 20:51:02.466949 (+ 260us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.466952 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.467015 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.467106 (+ 91us) server_negotiation.cc:227] Negotiated authn=JWT
0128 20:51:02.467663 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.468208 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:02.469235 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.469394 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:02.469499 (+ 105us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 20:51:02.489219 (+ 19720us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0128 20:51:02.489332 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:02.489598 (+ 266us) server_negotiation.cc:300] Negotiation successful
0128 20:51:02.489671 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":181,"threads_started":1}
I20260128 20:51:02.505542 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:02.500308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.4.183.190:41373)
0128 20:51:02.500496 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:02.500503 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:02.500728 (+ 225us) server_negotiation.cc:408] Connection header received
0128 20:51:02.500914 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:02.500916 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:02.500965 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:02.501058 (+ 93us) server_negotiation.cc:227] Negotiated authn=JWT
0128 20:51:02.501580 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.502431 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:02.503457 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.503693 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:02.503758 (+ 65us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 20:51:02.505037 (+ 1279us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0128 20:51:02.505398 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35220: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":80}
W20260128 20:51:02.505642 10647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35220: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260128 20:51:02.506425 10680 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:02.500179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:41373 (local address 127.0.0.1:35220)
0128 20:51:02.500631 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:02.500644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:02.500793 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:02.501074 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:02.501077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:02.501085 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0128 20:51:02.501387 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:02.501397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.502659 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:02.502663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:02.503306 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:02.503323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:02.503463 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:02.503504 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 20:51:02.505341 (+ 1837us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0128 20:51:02.505565 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.183.190:41373: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":170,"threads_started":1}
I20260128 20:51:17.516072 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:17.511321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.4.183.190:41373)
0128 20:51:17.511616 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:17.511619 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:17.511634 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:17.511692 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:17.511696 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:17.511747 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:17.511843 (+ 96us) server_negotiation.cc:227] Negotiated authn=JWT
0128 20:51:17.512511 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.513261 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:17.514265 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.514484 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:17.514543 (+ 59us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 20:51:17.515600 (+ 1057us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0128 20:51:17.515910 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60190: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":119,"threads_started":1}
W20260128 20:51:17.516153 10686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60190: JWT verification failed: token expired
W20260128 20:51:17.516387 10685 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:17.511181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:41373 (local address 127.0.0.1:60190)
0128 20:51:17.511490 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:17.511505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:17.511598 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:17.511905 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:17.511909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:17.511917 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0128 20:51:17.512146 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:17.512154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.513573 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:17.513580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:17.514099 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:17.514114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.514256 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:17.514294 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 20:51:17.515856 (+ 1562us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0128 20:51:17.516079 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.183.190:41373: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":144,"threads_started":1}
W20260128 20:51:17.520372 10686 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:17.519241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.4.183.190:41373)
0128 20:51:17.519408 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:17.519412 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:17.519654 (+ 242us) server_negotiation.cc:408] Connection header received
0128 20:51:17.519797 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:17.519800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:17.519857 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:17.519959 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:17.520260 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60200: BlockingRecv error: recv got EOF from 127.0.0.1:60200 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260128 20:51:17.524444 10696 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.4.183.190:41373
W20260128 20:51:17.525336 10686 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:17.524143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.4.183.190:41373)
0128 20:51:17.524306 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:17.524310 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:17.524420 (+ 110us) server_negotiation.cc:408] Connection header received
0128 20:51:17.524656 (+ 236us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:17.524659 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:17.524724 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:17.524813 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:17.525228 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60210: BlockingRecv error: recv got EOF from 127.0.0.1:60210 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260128 20:51:17.525507 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10577
2026-01-28T20:51:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15395 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260128 20:51:17.567694 4830 mini_oidc.cc:150] Starting JWKS server
I20260128 20:51:17.568652 4830 webserver.cc:492] Webserver started at https://127.0.0.1:38623/ using document root <none> and password file <none>
I20260128 20:51:17.568744 4830 mini_oidc.cc:186] Starting OIDC Discovery server
I20260128 20:51:17.568938 4830 webserver.cc:492] Webserver started at http://127.0.0.1:33157/ using document root <none> and password file <none>
2026-01-28T20:51:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:17Z Disabled control of system clock
I20260128 20:51:17.578557 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:33311
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:33311
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/testchainca.cert with env {}
W20260128 20:51:17.699781 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:17.700110 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:17.700169 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:17.703866 10702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:17.703969 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:17.703987 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:17.704007 10702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:17.704026 10702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 20:51:17.704046 10702 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 20:51:17.704059 10702 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38623/jwks.json
I20260128 20:51:17.708971 10702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:33311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:17.710217 10702 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:17.711129 10702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:17.718727 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:17.718727 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:17.718757 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:17.719151 10702 server_base.cc:1060] running on GCE node
I20260128 20:51:17.719741 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:17.720852 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:17.722081 10702 hybrid_clock.cc:648] HybridClock initialized: now 1769633477722050 us; error 50 us; skew 500 ppm
I20260128 20:51:17.724615 10702 webserver.cc:492] Webserver started at http://127.4.183.190:39605/ using document root <none> and password file <none>
I20260128 20:51:17.725430 10702 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:17.725512 10702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:17.725766 10702 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:17.727787 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "4f703cbbfa8a40a0aaebf50565134c12"
format_stamp: "Formatted at 2026-01-28 20:51:17 on dist-test-slave-hss7"
server_key: "d00fc4b6ed9fd0af84e04c5f32d6e7ab"
server_key_iv: "708becdc97adad016c0546d43febdf05"
server_key_version: "encryptionkey@0"
I20260128 20:51:17.728396 10702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "4f703cbbfa8a40a0aaebf50565134c12"
format_stamp: "Formatted at 2026-01-28 20:51:17 on dist-test-slave-hss7"
server_key: "d00fc4b6ed9fd0af84e04c5f32d6e7ab"
server_key_iv: "708becdc97adad016c0546d43febdf05"
server_key_version: "encryptionkey@0"
I20260128 20:51:17.733225 10702 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260128 20:51:17.736430 10716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:17.737880 10702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:17.738080 10702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "4f703cbbfa8a40a0aaebf50565134c12"
format_stamp: "Formatted at 2026-01-28 20:51:17 on dist-test-slave-hss7"
server_key: "d00fc4b6ed9fd0af84e04c5f32d6e7ab"
server_key_iv: "708becdc97adad016c0546d43febdf05"
server_key_version: "encryptionkey@0"
I20260128 20:51:17.738219 10702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:17.752321 10702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:17.753279 10702 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:17.753513 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:17.765398 10768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:33311 every 8 connection(s)
I20260128 20:51:17.765394 10702 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:33311
I20260128 20:51:17.766714 10702 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:17.770124 10769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:17.776553 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10702
I20260128 20:51:17.776714 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:17.776876 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12: Bootstrap starting.
I20260128 20:51:17.776980 4830 external_mini_cluster.cc:1468] Setting key fa25ee9cc7b5fa85aeca667518fccd81
I20260128 20:51:17.779848 10769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:17.780727 10769 log.cc:826] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:17.782943 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12: No bootstrap required, opened a new log
I20260128 20:51:17.785940 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:17.778440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.4.183.190:33311)
0128 20:51:17.779322 (+ 882us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:17.779333 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:17.779377 (+ 44us) server_negotiation.cc:408] Connection header received
0128 20:51:17.780110 (+ 733us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:17.780135 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:17.780556 (+ 421us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:17.780964 (+ 408us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:17.781565 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.782325 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:17.783081 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.783409 (+ 328us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:17.783966 (+ 557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:17.783990 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:17.784002 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:17.784423 (+ 421us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:17.784475 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:17.784491 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:17.784637 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:17.784833 (+ 196us) server_negotiation.cc:300] Negotiation successful
0128 20:51:17.785112 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":213,"threads_started":1}
I20260128 20:51:17.786345 10769 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f703cbbfa8a40a0aaebf50565134c12" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33311 } }
I20260128 20:51:17.786639 10769 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:17.786715 10769 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f703cbbfa8a40a0aaebf50565134c12, State: Initialized, Role: FOLLOWER
I20260128 20:51:17.787186 10769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f703cbbfa8a40a0aaebf50565134c12" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33311 } }
I20260128 20:51:17.787355 10769 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:17.787436 10769 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:17.787559 10769 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:17.788744 10769 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f703cbbfa8a40a0aaebf50565134c12" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33311 } }
I20260128 20:51:17.789196 10769 leader_election.cc:304] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f703cbbfa8a40a0aaebf50565134c12; no voters:
I20260128 20:51:17.789505 4830 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 20:51:17.789638 10769 leader_election.cc:290] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:17.789816 10774 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:17.790045 10774 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [term 1 LEADER]: Becoming Leader. State: Replica: 4f703cbbfa8a40a0aaebf50565134c12, State: Running, Role: LEADER
I20260128 20:51:17.790393 10774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f703cbbfa8a40a0aaebf50565134c12" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33311 } }
I20260128 20:51:17.791060 10769 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:17.793380 10776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f703cbbfa8a40a0aaebf50565134c12. Latest consensus state: current_term: 1 leader_uuid: "4f703cbbfa8a40a0aaebf50565134c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f703cbbfa8a40a0aaebf50565134c12" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33311 } } }
I20260128 20:51:17.793574 10776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:17.794028 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f703cbbfa8a40a0aaebf50565134c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f703cbbfa8a40a0aaebf50565134c12" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 33311 } } }
I20260128 20:51:17.794139 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12 [sys.catalog]: This master's current role is: LEADER
W20260128 20:51:17.798549 10788 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:17.798689 10788 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:17.798784 10789 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:17.799708 10789 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:17.806584 10789 catalog_manager.cc:1357] Generated new cluster ID: 6f035be250b24b2098e557cace7b4b8d
I20260128 20:51:17.806687 10789 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:17.816412 10789 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:17.818041 10789 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:17.839573 10789 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4f703cbbfa8a40a0aaebf50565134c12: Generated new TSK 0
I20260128 20:51:17.840480 10789 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260128 20:51:17.885279 10799 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260128 20:51:17.885377 10799 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260128 20:51:17.886130 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:17.860589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.4.183.190:33311)
0128 20:51:17.860747 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:17.860751 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:17.860851 (+ 100us) server_negotiation.cc:408] Connection header received
0128 20:51:17.861117 (+ 266us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:17.861120 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:17.861263 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:17.861365 (+ 102us) server_negotiation.cc:227] Negotiated authn=JWT
0128 20:51:17.861899 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.862457 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:17.863505 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.863691 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:17.863814 (+ 123us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 20:51:17.885704 (+ 21890us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 20:51:17.885998 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260128 20:51:17.886245 10772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260128 20:51:17.886349 10798 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:17.860312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:33311 (local address 127.0.0.1:42614)
0128 20:51:17.860723 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:17.860741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:17.860883 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:17.861391 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:17.861395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:17.861405 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0128 20:51:17.861726 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:17.861737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.862592 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:17.862596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:17.863305 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:17.863318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:17.863481 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:17.863515 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 20:51:17.885964 (+ 22449us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 20:51:17.886145 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.183.190:33311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":195,"threads_started":1}
I20260128 20:51:17.887046 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10702
2026-01-28T20:51:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (360 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260128 20:51:17.927879 4830 mini_oidc.cc:150] Starting JWKS server
I20260128 20:51:17.928885 4830 webserver.cc:492] Webserver started at https://127.0.0.1:41085/ using document root <none> and password file <none>
I20260128 20:51:17.929044 4830 mini_oidc.cc:186] Starting OIDC Discovery server
I20260128 20:51:17.929402 4830 webserver.cc:492] Webserver started at http://127.0.0.1:38763/ using document root <none> and password file <none>
2026-01-28T20:51:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:17Z Disabled control of system clock
I20260128 20:51:17.938874 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39193
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:39193
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41085/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260128 20:51:18.058622 10805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:18.058910 10805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:18.058964 10805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:18.062608 10805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:18.062704 10805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:18.062721 10805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:18.062741 10805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:18.062758 10805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 20:51:18.062777 10805 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 20:51:18.062791 10805 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41085/jwks.json
I20260128 20:51:18.067721 10805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:39193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41085/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:18.068890 10805 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:18.070170 10805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:18.076598 10810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:18.076689 10805 server_base.cc:1060] running on GCE node
W20260128 20:51:18.076848 10813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:18.076858 10811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:18.077772 10805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:18.078909 10805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:18.080128 10805 hybrid_clock.cc:648] HybridClock initialized: now 1769633478080101 us; error 51 us; skew 500 ppm
I20260128 20:51:18.082643 10805 webserver.cc:492] Webserver started at http://127.4.183.190:39101/ using document root <none> and password file <none>
I20260128 20:51:18.083355 10805 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:18.083431 10805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:18.083681 10805 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:18.086194 10805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "a721da92332b4706900f070f400691c1"
format_stamp: "Formatted at 2026-01-28 20:51:18 on dist-test-slave-hss7"
server_key: "e25b9112fe4a9c082e51016550389ce9"
server_key_iv: "113150d4c8aa61e23e1e087274690442"
server_key_version: "encryptionkey@0"
I20260128 20:51:18.086827 10805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "a721da92332b4706900f070f400691c1"
format_stamp: "Formatted at 2026-01-28 20:51:18 on dist-test-slave-hss7"
server_key: "e25b9112fe4a9c082e51016550389ce9"
server_key_iv: "113150d4c8aa61e23e1e087274690442"
server_key_version: "encryptionkey@0"
I20260128 20:51:18.091158 10805 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260128 20:51:18.094273 10819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:18.096004 10805 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:18.096340 10805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "a721da92332b4706900f070f400691c1"
format_stamp: "Formatted at 2026-01-28 20:51:18 on dist-test-slave-hss7"
server_key: "e25b9112fe4a9c082e51016550389ce9"
server_key_iv: "113150d4c8aa61e23e1e087274690442"
server_key_version: "encryptionkey@0"
I20260128 20:51:18.096537 10805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:18.114495 10805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:18.115271 10805 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:18.115485 10805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:18.124620 10871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:39193 every 8 connection(s)
I20260128 20:51:18.124614 10805 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:39193
I20260128 20:51:18.126255 10805 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:18.126699 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10805
I20260128 20:51:18.126847 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:18.127164 4830 external_mini_cluster.cc:1468] Setting key c871bb38d460b622047b2b4f7a12b6c3
I20260128 20:51:18.130169 10872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:18.136536 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:18.128475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.4.183.190:39193)
0128 20:51:18.129273 (+ 798us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:18.129287 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:18.129338 (+ 51us) server_negotiation.cc:408] Connection header received
0128 20:51:18.130028 (+ 690us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:18.130047 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:18.130518 (+ 471us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:18.131037 (+ 519us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:18.131634 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.132886 (+ 1252us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:18.133812 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.134188 (+ 376us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:18.134708 (+ 520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:18.134773 (+ 65us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:18.134787 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:18.135222 (+ 435us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:18.135271 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:18.135287 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:18.135420 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:18.135614 (+ 194us) server_negotiation.cc:300] Negotiation successful
0128 20:51:18.135883 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":151,"threads_started":1}
I20260128 20:51:18.137472 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1: Bootstrap starting.
I20260128 20:51:18.139703 4830 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 20:51:18.140044 10872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:18.140944 10872 log.cc:826] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:18.143167 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1: No bootstrap required, opened a new log
I20260128 20:51:18.146098 10872 raft_consensus.cc:359] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a721da92332b4706900f070f400691c1" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39193 } }
I20260128 20:51:18.146402 10872 raft_consensus.cc:385] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:18.146512 10872 raft_consensus.cc:740] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a721da92332b4706900f070f400691c1, State: Initialized, Role: FOLLOWER
I20260128 20:51:18.147006 10872 consensus_queue.cc:260] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a721da92332b4706900f070f400691c1" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39193 } }
I20260128 20:51:18.147187 10872 raft_consensus.cc:399] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:18.147271 10872 raft_consensus.cc:493] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:18.147396 10872 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:18.148418 10872 raft_consensus.cc:515] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a721da92332b4706900f070f400691c1" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39193 } }
I20260128 20:51:18.148903 10872 leader_election.cc:304] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a721da92332b4706900f070f400691c1; no voters:
I20260128 20:51:18.149349 10872 leader_election.cc:290] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:18.149537 10877 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:18.149808 10877 raft_consensus.cc:697] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [term 1 LEADER]: Becoming Leader. State: Replica: a721da92332b4706900f070f400691c1, State: Running, Role: LEADER
I20260128 20:51:18.150157 10877 consensus_queue.cc:237] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a721da92332b4706900f070f400691c1" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39193 } }
I20260128 20:51:18.150668 10872 sys_catalog.cc:565] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:18.152519 10879 sys_catalog.cc:455] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a721da92332b4706900f070f400691c1. Latest consensus state: current_term: 1 leader_uuid: "a721da92332b4706900f070f400691c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a721da92332b4706900f070f400691c1" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39193 } } }
I20260128 20:51:18.152787 10879 sys_catalog.cc:458] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:18.152551 10878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a721da92332b4706900f070f400691c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a721da92332b4706900f070f400691c1" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39193 } } }
I20260128 20:51:18.152935 10878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:18.153796 10886 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:18.158185 10886 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:18.165799 10886 catalog_manager.cc:1357] Generated new cluster ID: 885164dd5c6a4a55b9445a6766390ab2
I20260128 20:51:18.165910 10886 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:18.216141 10886 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:18.217885 10886 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:18.230448 10886 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a721da92332b4706900f070f400691c1: Generated new TSK 0
I20260128 20:51:18.231485 10886 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260128 20:51:18.276278 10902 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260128 20:51:18.276384 10902 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260128 20:51:18.278759 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:18.252933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.4.183.190:39193)
0128 20:51:18.253107 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:18.253113 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:18.253315 (+ 202us) server_negotiation.cc:408] Connection header received
0128 20:51:18.253496 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:18.253501 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:18.253578 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:18.253713 (+ 135us) server_negotiation.cc:227] Negotiated authn=JWT
0128 20:51:18.254183 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.254762 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:18.255482 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.255709 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:18.255828 (+ 119us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 20:51:18.278082 (+ 22254us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 20:51:18.278633 (+ 551us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42766: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260128 20:51:18.278739 10901 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:18.252746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39193 (local address 127.0.0.1:42766)
0128 20:51:18.253211 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:18.253232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:18.253347 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:18.253737 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:18.253741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:18.253749 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0128 20:51:18.254038 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:18.254048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.254896 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:18.254902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:18.255350 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:18.255360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.255530 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:18.255565 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 20:51:18.278359 (+ 22794us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 20:51:18.278511 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.183.190:39193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":141,"threads_started":1}
W20260128 20:51:18.278852 10875 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42766: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260128 20:51:18.279743 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10805
2026-01-28T20:51:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (393 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-01-28T20:51:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:18Z Disabled control of system clock
I20260128 20:51:18.330116 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:37783
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:37783
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/testchainca.cert with env {}
W20260128 20:51:18.454190 10906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:18.454474 10906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:18.454536 10906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:18.458215 10906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:18.458312 10906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:18.458329 10906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:18.458349 10906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:18.458367 10906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 20:51:18.458386 10906 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 20:51:18.458401 10906 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260128 20:51:18.463456 10906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:37783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:37783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:18.464746 10906 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:18.465724 10906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:18.471966 10911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:18.471966 10914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:18.471975 10912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:18.472280 10906 server_base.cc:1060] running on GCE node
I20260128 20:51:18.472978 10906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:18.474165 10906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:18.475515 10906 hybrid_clock.cc:648] HybridClock initialized: now 1769633478475467 us; error 55 us; skew 500 ppm
I20260128 20:51:18.478541 10906 webserver.cc:492] Webserver started at http://127.4.183.190:45605/ using document root <none> and password file <none>
I20260128 20:51:18.479269 10906 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:18.479378 10906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:18.479632 10906 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:18.481745 10906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "d11d5ebd02584faaa01642a890fb9ffa"
format_stamp: "Formatted at 2026-01-28 20:51:18 on dist-test-slave-hss7"
server_key: "9e2f7837ee629754a39794b2c42073b4"
server_key_iv: "46c9988e152bbac900d7d560e06fe1d9"
server_key_version: "encryptionkey@0"
I20260128 20:51:18.482326 10906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "d11d5ebd02584faaa01642a890fb9ffa"
format_stamp: "Formatted at 2026-01-28 20:51:18 on dist-test-slave-hss7"
server_key: "9e2f7837ee629754a39794b2c42073b4"
server_key_iv: "46c9988e152bbac900d7d560e06fe1d9"
server_key_version: "encryptionkey@0"
I20260128 20:51:18.486411 10906 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:51:18.489420 10920 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:18.490768 10906 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260128 20:51:18.490985 10906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "d11d5ebd02584faaa01642a890fb9ffa"
format_stamp: "Formatted at 2026-01-28 20:51:18 on dist-test-slave-hss7"
server_key: "9e2f7837ee629754a39794b2c42073b4"
server_key_iv: "46c9988e152bbac900d7d560e06fe1d9"
server_key_version: "encryptionkey@0"
I20260128 20:51:18.491125 10906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:18.509678 10906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:18.510443 10906 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:18.510667 10906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:18.518847 10906 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:37783
I20260128 20:51:18.518880 10972 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:37783 every 8 connection(s)
I20260128 20:51:18.520011 10906 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:18.523265 10973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:18.527814 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 10906
I20260128 20:51:18.527948 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:18.528244 4830 external_mini_cluster.cc:1468] Setting key b405521dc448bd7e89bdbe98ee0a599e
I20260128 20:51:18.530422 10973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa: Bootstrap starting.
I20260128 20:51:18.533128 10973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:18.534163 10973 log.cc:826] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:18.536561 10973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa: No bootstrap required, opened a new log
I20260128 20:51:18.537618 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:18.529840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.4.183.190:37783)
0128 20:51:18.530837 (+ 997us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:18.530848 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:18.530889 (+ 41us) server_negotiation.cc:408] Connection header received
0128 20:51:18.531494 (+ 605us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:18.531523 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:18.531917 (+ 394us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:18.532257 (+ 340us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:18.532957 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.533925 (+ 968us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:18.534910 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.535249 (+ 339us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:18.535732 (+ 483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:18.535756 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:18.535769 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:18.536176 (+ 407us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:18.536226 (+ 50us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:18.536241 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:18.536371 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:18.536587 (+ 216us) server_negotiation.cc:300] Negotiation successful
0128 20:51:18.536884 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":161,"threads_started":1}
I20260128 20:51:18.540158 10973 raft_consensus.cc:359] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11d5ebd02584faaa01642a890fb9ffa" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 37783 } }
I20260128 20:51:18.540498 10973 raft_consensus.cc:385] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:18.540568 10973 raft_consensus.cc:740] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d5ebd02584faaa01642a890fb9ffa, State: Initialized, Role: FOLLOWER
I20260128 20:51:18.541121 10973 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11d5ebd02584faaa01642a890fb9ffa" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 37783 } }
I20260128 20:51:18.541260 4830 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 20:51:18.541337 10973 raft_consensus.cc:399] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:18.541411 10973 raft_consensus.cc:493] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:18.541527 10973 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:18.542738 10973 raft_consensus.cc:515] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11d5ebd02584faaa01642a890fb9ffa" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 37783 } }
I20260128 20:51:18.543146 10973 leader_election.cc:304] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11d5ebd02584faaa01642a890fb9ffa; no voters:
I20260128 20:51:18.543565 10973 leader_election.cc:290] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:18.543831 10978 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:18.544075 10978 raft_consensus.cc:697] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [term 1 LEADER]: Becoming Leader. State: Replica: d11d5ebd02584faaa01642a890fb9ffa, State: Running, Role: LEADER
I20260128 20:51:18.544423 10978 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11d5ebd02584faaa01642a890fb9ffa" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 37783 } }
I20260128 20:51:18.544862 10973 sys_catalog.cc:565] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:18.547031 10979 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11d5ebd02584faaa01642a890fb9ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11d5ebd02584faaa01642a890fb9ffa" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 37783 } } }
I20260128 20:51:18.547204 10979 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:18.547528 10980 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11d5ebd02584faaa01642a890fb9ffa. Latest consensus state: current_term: 1 leader_uuid: "d11d5ebd02584faaa01642a890fb9ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11d5ebd02584faaa01642a890fb9ffa" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 37783 } } }
I20260128 20:51:18.547622 10980 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa [sys.catalog]: This master's current role is: LEADER
W20260128 20:51:18.551023 10993 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:18.551122 10993 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:18.551223 10992 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:18.552070 10992 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:18.558569 10992 catalog_manager.cc:1357] Generated new cluster ID: 1d0542ec389d481e95ccbc224af608b6
I20260128 20:51:18.558683 10992 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:18.570905 10992 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:18.572033 10992 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:18.580834 10992 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d11d5ebd02584faaa01642a890fb9ffa: Generated new TSK 0
I20260128 20:51:18.581897 10992 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:51:18.654386 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:18.597734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.4.183.190:37783)
0128 20:51:18.597910 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:18.597917 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:18.598084 (+ 167us) server_negotiation.cc:408] Connection header received
0128 20:51:18.598278 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:18.598281 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:18.598343 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:18.598439 (+ 96us) server_negotiation.cc:227] Negotiated authn=JWT
0128 20:51:18.599013 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.599582 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:18.600620 (+ 1038us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.600844 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:18.600983 (+ 139us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 20:51:18.653480 (+ 52497us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0128 20:51:18.653987 (+ 507us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36020: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260128 20:51:18.654518 10976 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36020: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260128 20:51:18.654585 11002 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:18.597624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:37783 (local address 127.0.0.1:36020)
0128 20:51:18.597974 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:18.597992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:18.598131 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:18.598471 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:18.598475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:18.598483 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0128 20:51:18.598845 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:18.598859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.599727 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:18.599732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:18.600423 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:18.600436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:18.600611 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:18.600652 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 20:51:18.653888 (+ 53236us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0128 20:51:18.654338 (+ 450us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.183.190:37783: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":157,"threads_started":1}
I20260128 20:51:18.655378 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 10906
2026-01-28T20:51:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (354 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (138 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (138 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:51:18 dist-test-slave-hss7 krb5kdc[11014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:51:19 dist-test-slave-hss7 krb5kdc[11014](info): set up 2 sockets
Jan 28 20:51:19 dist-test-slave-hss7 krb5kdc[11014](info): commencing operation
krb5kdc: starting...
W20260128 20:51:20.967700 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:51:21 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:51:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:21.145951 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39147
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:39147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:21.267838 11030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:21.268185 11030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:21.268281 11030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:21.272091 11030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:21.272246 11030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:21.272290 11030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:21.272333 11030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:21.272404 11030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:21.277716 11030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:39147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:21.279057 11030 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:21.280043 11030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:21.286891 11035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:21.286875 11036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:21.287053 11030 server_base.cc:1060] running on GCE node
W20260128 20:51:21.286875 11038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:21.287849 11030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:21.288978 11030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:21.290210 11030 hybrid_clock.cc:648] HybridClock initialized: now 1769633481290193 us; error 39 us; skew 500 ppm
Jan 28 20:51:21 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:21.295238 11030 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:51:21.296967 11030 webserver.cc:492] Webserver started at http://127.4.183.190:38085/ using document root <none> and password file <none>
I20260128 20:51:21.297829 11030 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:21.297925 11030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:21.298203 11030 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:21.300431 11030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "db8fab5e21e54e139c14753a8dfdf26f"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "92a790e1215539bfe7838eca3587afa4"
server_key_iv: "06073aa6fb8dca5f6008f56a42579e44"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.301003 11030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "db8fab5e21e54e139c14753a8dfdf26f"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "92a790e1215539bfe7838eca3587afa4"
server_key_iv: "06073aa6fb8dca5f6008f56a42579e44"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.305292 11030 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260128 20:51:21.308562 11045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:21.310371 11030 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260128 20:51:21.310751 11030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "db8fab5e21e54e139c14753a8dfdf26f"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "92a790e1215539bfe7838eca3587afa4"
server_key_iv: "06073aa6fb8dca5f6008f56a42579e44"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.310907 11030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:21.327405 11030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:21.336086 11030 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:21.336295 11030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:21.346418 11030 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:39147
I20260128 20:51:21.346462 11097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:39147 every 8 connection(s)
I20260128 20:51:21.347687 11030 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:21.351182 11098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:21.354193 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11030
I20260128 20:51:21.354372 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:21.354710 4830 external_mini_cluster.cc:1468] Setting key b88dbacb0b7f1395cda9a4e01fad858e
I20260128 20:51:21.358853 11098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f: Bootstrap starting.
I20260128 20:51:21.361465 11098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:21.362474 11098 log.cc:826] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f: Log is configured to *not* fsync() on all Append() calls
Jan 28 20:51:21 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:21.366241 11098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f: No bootstrap required, opened a new log
I20260128 20:51:21.370155 11098 raft_consensus.cc:359] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8fab5e21e54e139c14753a8dfdf26f" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39147 } }
I20260128 20:51:21.370507 11098 raft_consensus.cc:385] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:21.370563 11098 raft_consensus.cc:740] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8fab5e21e54e139c14753a8dfdf26f, State: Initialized, Role: FOLLOWER
I20260128 20:51:21.371290 11098 consensus_queue.cc:260] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8fab5e21e54e139c14753a8dfdf26f" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39147 } }
I20260128 20:51:21.371529 11098 raft_consensus.cc:399] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:21.371613 11098 raft_consensus.cc:493] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:21.371934 11098 raft_consensus.cc:3060] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:21.373319 11098 raft_consensus.cc:515] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8fab5e21e54e139c14753a8dfdf26f" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39147 } }
I20260128 20:51:21.373673 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:21.356623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.4.183.190:39147)
0128 20:51:21.357373 (+ 750us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:21.357384 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:21.357426 (+ 42us) server_negotiation.cc:408] Connection header received
0128 20:51:21.358041 (+ 615us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:21.358063 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:21.358405 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:21.358845 (+ 440us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:21.359904 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:21.360706 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:21.361714 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:21.362091 (+ 377us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:21.365606 (+ 3515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:21.365635 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:21.365653 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:21.365719 (+ 66us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:21.368515 (+ 2796us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:21.369511 (+ 996us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:21.369519 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:21.369526 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:21.369666 (+ 140us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:21.370311 (+ 645us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:21.370316 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:21.370318 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:21.371128 (+ 810us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:21.371632 (+ 504us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:21.371918 (+ 286us) server_negotiation.cc:300] Negotiation successful
0128 20:51:21.372354 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":219,"threads_started":1}
I20260128 20:51:21.374418 11098 leader_election.cc:304] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db8fab5e21e54e139c14753a8dfdf26f; no voters:
I20260128 20:51:21.374950 11098 leader_election.cc:290] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:21.375273 11103 raft_consensus.cc:2804] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:21.376089 11103 raft_consensus.cc:697] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [term 1 LEADER]: Becoming Leader. State: Replica: db8fab5e21e54e139c14753a8dfdf26f, State: Running, Role: LEADER
I20260128 20:51:21.377218 11098 sys_catalog.cc:565] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:21.377218 11103 consensus_queue.cc:237] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8fab5e21e54e139c14753a8dfdf26f" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39147 } }
I20260128 20:51:21.380017 11104 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader db8fab5e21e54e139c14753a8dfdf26f. Latest consensus state: current_term: 1 leader_uuid: "db8fab5e21e54e139c14753a8dfdf26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8fab5e21e54e139c14753a8dfdf26f" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39147 } } }
I20260128 20:51:21.380050 11105 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db8fab5e21e54e139c14753a8dfdf26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8fab5e21e54e139c14753a8dfdf26f" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39147 } } }
I20260128 20:51:21.380251 11104 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:21.380252 11105 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f [sys.catalog]: This master's current role is: LEADER
W20260128 20:51:21.383186 11117 catalog_manager.cc:1568] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:21.383351 11117 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:21.383445 11118 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:21.384339 11118 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:21.391126 11118 catalog_manager.cc:1357] Generated new cluster ID: 9cdac10cd6c247df86d6a50523e5f9d6
I20260128 20:51:21.391249 11118 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:21.406296 11118 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:21.407754 11118 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:21.424279 11118 catalog_manager.cc:6027] T 00000000000000000000000000000000 P db8fab5e21e54e139c14753a8dfdf26f: Generated new TSK 0
I20260128 20:51:21.425654 11118 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:21.498734 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39147
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:21.643460 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:21.643801 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:21.643929 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:21.648360 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:21.648465 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:21.648554 11126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:21.653673 11126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:21.654917 11126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:21.655889 11126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:21.664449 11131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:21.664558 11126 server_base.cc:1060] running on GCE node
W20260128 20:51:21.664449 11132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:21.664449 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:21.665614 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:21.666601 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:21.667992 11126 hybrid_clock.cc:648] HybridClock initialized: now 1769633481667932 us; error 86 us; skew 500 ppm
Jan 28 20:51:21 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:21.671726 11126 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:51:21.674177 11126 webserver.cc:492] Webserver started at http://127.4.183.129:40131/ using document root <none> and password file <none>
I20260128 20:51:21.675055 11126 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:21.675154 11126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:21.675683 11126 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:21.678004 11126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "46b09eebe311405c88595f83ece82a02"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "c234885d309d7bd0c1aa9f5f9ae7f40b"
server_key_iv: "ded8435def8cb8066baa9eedab971149"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.678678 11126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "46b09eebe311405c88595f83ece82a02"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "c234885d309d7bd0c1aa9f5f9ae7f40b"
server_key_iv: "ded8435def8cb8066baa9eedab971149"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.683233 11126 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260128 20:51:21.686658 11141 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:21.688046 11126 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:21.688236 11126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "46b09eebe311405c88595f83ece82a02"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "c234885d309d7bd0c1aa9f5f9ae7f40b"
server_key_iv: "ded8435def8cb8066baa9eedab971149"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.688372 11126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:21.712188 11126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:21.716145 11126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:21.716408 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:21.717099 11126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:21.718313 11126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:21.718406 11126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:21.718485 11126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:21.718539 11126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:21.729508 11126 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:39185
I20260128 20:51:21.729513 11254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:39185 every 8 connection(s)
I20260128 20:51:21.730908 11126 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:21.737869 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11126
I20260128 20:51:21.737991 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:21.738255 4830 external_mini_cluster.cc:1468] Setting key e81ea2771ab751faeb80b575b0cdde21
Jan 28 20:51:21 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:21.746920 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:21.733302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:54311 (local address 127.4.183.190:39147)
0128 20:51:21.733466 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:21.733470 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:21.734416 (+ 946us) server_negotiation.cc:408] Connection header received
0128 20:51:21.735388 (+ 972us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:21.735392 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:21.735451 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:21.735574 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:21.737748 (+ 2174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:21.738436 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:21.739389 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:21.739700 (+ 311us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:21.742248 (+ 2548us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:21.742282 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:21.742288 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:21.742333 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:21.744430 (+ 2097us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:21.745122 (+ 692us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:21.745125 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:21.745127 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:21.745255 (+ 128us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:21.745601 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:21.745605 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:21.745607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:21.745801 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:21.745920 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:21.746562 (+ 642us) server_negotiation.cc:300] Negotiation successful
0128 20:51:21.746730 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 20:51:21.747669 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:21.733582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39147 (local address 127.4.183.129:54311)
0128 20:51:21.734226 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:21.734272 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:21.735127 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:21.735786 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:21.735804 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:21.736278 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:21.737448 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:21.737469 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:21.738614 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:21.738619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:21.739199 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:21.739209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:21.739426 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:21.740187 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:21.740234 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:21.742019 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:21.744632 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:21.744639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:21.744652 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:21.744975 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:21.745359 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:21.745365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:21.745371 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:21.745487 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:21.745923 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:21.745934 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:21.746250 (+ 316us) client_negotiation.cc:769] Sending connection context
0128 20:51:21.746503 (+ 253us) client_negotiation.cc:240] Negotiation successful
0128 20:51:21.746851 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
I20260128 20:51:21.749091 11255 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39147
I20260128 20:51:21.749420 11255 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:21.750072 11255 heartbeater.cc:507] Master 127.4.183.190:39147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:51:21.751994 11062 ts_manager.cc:194] Registered new tserver with Master: 46b09eebe311405c88595f83ece82a02 (127.4.183.129:39185)
I20260128 20:51:21.753578 11062 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:54311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:21.797894 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39147
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:21.927217 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:21.927472 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:21.927544 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:21.931432 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:21.931542 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:21.931631 11262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:21.936630 11262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:21.937956 11262 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:21.938848 11262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:21.946338 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:21.946338 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:21.946338 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:21.946669 11262 server_base.cc:1060] running on GCE node
I20260128 20:51:21.947150 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:21.947842 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:21.949079 11262 hybrid_clock.cc:648] HybridClock initialized: now 1769633481949044 us; error 53 us; skew 500 ppm
Jan 28 20:51:21 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:21.952466 11262 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:51:21.954068 11262 webserver.cc:492] Webserver started at http://127.4.183.130:43969/ using document root <none> and password file <none>
I20260128 20:51:21.954716 11262 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:21.954785 11262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:21.954979 11262 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:21.956918 11262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "edb2e0f55ccb4721bf0197e186e001ec"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "f43f577c4af63c73a9fecae14909d917"
server_key_iv: "0cc08685508bb0b07e29331291217541"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.957551 11262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "edb2e0f55ccb4721bf0197e186e001ec"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "f43f577c4af63c73a9fecae14909d917"
server_key_iv: "0cc08685508bb0b07e29331291217541"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.961611 11262 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:51:21.964459 11277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:21.965951 11262 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:21.966092 11262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "edb2e0f55ccb4721bf0197e186e001ec"
format_stamp: "Formatted at 2026-01-28 20:51:21 on dist-test-slave-hss7"
server_key: "f43f577c4af63c73a9fecae14909d917"
server_key_iv: "0cc08685508bb0b07e29331291217541"
server_key_version: "encryptionkey@0"
I20260128 20:51:21.966198 11262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:21.985967 11262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:21.989465 11262 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:21.989648 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:21.990216 11262 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:21.991186 11262 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:21.991238 11262 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:21.991282 11262 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:21.991297 11262 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:22.002087 11262 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:40229
I20260128 20:51:22.002168 11390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:40229 every 8 connection(s)
I20260128 20:51:22.003252 11262 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:22.005822 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11262
I20260128 20:51:22.005961 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:22.006197 4830 external_mini_cluster.cc:1468] Setting key de157d5660dc165983d4e0cb6323f33d
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:51:22.025791 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.005456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:39255 (local address 127.4.183.190:39147)
0128 20:51:22.005681 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.005686 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.006514 (+ 828us) server_negotiation.cc:408] Connection header received
0128 20:51:22.007667 (+ 1153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.007674 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.007735 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.007827 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.010031 (+ 2204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.010570 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.011668 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.011905 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.019260 (+ 7355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:22.019474 (+ 214us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:22.019491 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:22.019578 (+ 87us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:22.022918 (+ 3340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.023847 (+ 929us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.023854 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.023859 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.023943 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.024366 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.024369 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.024371 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.024577 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:22.024707 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:22.025411 (+ 704us) server_negotiation.cc:300] Negotiation successful
0128 20:51:22.025590 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260128 20:51:22.026561 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.005736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39147 (local address 127.4.183.130:39255)
0128 20:51:22.006328 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:22.006378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:22.007332 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:22.008030 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:22.008041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:22.008477 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:22.009774 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:22.009803 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.010750 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.010757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:22.011437 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:22.011452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.011772 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.012504 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:22.012566 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:22.016455 (+ 3889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:22.023177 (+ 6722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:22.023186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:22.023202 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:22.023646 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:22.024081 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:22.024084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:22.024086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:22.024206 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:22.024708 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:22.024719 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:22.025030 (+ 311us) client_negotiation.cc:769] Sending connection context
0128 20:51:22.025349 (+ 319us) client_negotiation.cc:240] Negotiation successful
0128 20:51:22.025712 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":157,"threads_started":1}
I20260128 20:51:22.027900 11391 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39147
I20260128 20:51:22.028189 11391 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:22.028831 11391 heartbeater.cc:507] Master 127.4.183.190:39147 requested a full tablet report, sending...
I20260128 20:51:22.030723 11062 ts_manager.cc:194] Registered new tserver with Master: edb2e0f55ccb4721bf0197e186e001ec (127.4.183.130:40229)
I20260128 20:51:22.031754 11062 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:39255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:22.079298 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39147
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:22.220113 11398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:22.220772 11398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:22.220880 11398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:22.225435 11398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:22.225553 11398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:22.225656 11398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:22.230803 11398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:22.232033 11398 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:22.232928 11398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:22.240911 11403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:22.240933 11404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:22.240911 11406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:22.241398 11398 server_base.cc:1060] running on GCE node
I20260128 20:51:22.242053 11398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:22.242770 11398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:22.243980 11398 hybrid_clock.cc:648] HybridClock initialized: now 1769633482243968 us; error 32 us; skew 500 ppm
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.247366 11398 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:51:22.248744 11398 webserver.cc:492] Webserver started at http://127.4.183.131:41689/ using document root <none> and password file <none>
I20260128 20:51:22.249486 11398 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:22.249580 11398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:22.249836 11398 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:22.251793 11398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "f8680c9dd37f419089d7388e87429646"
format_stamp: "Formatted at 2026-01-28 20:51:22 on dist-test-slave-hss7"
server_key: "9c4290b0beb248bafa1088ef93bb0875"
server_key_iv: "bd40f4e0d9e0f01655940f9220700484"
server_key_version: "encryptionkey@0"
I20260128 20:51:22.252352 11398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "f8680c9dd37f419089d7388e87429646"
format_stamp: "Formatted at 2026-01-28 20:51:22 on dist-test-slave-hss7"
server_key: "9c4290b0beb248bafa1088ef93bb0875"
server_key_iv: "bd40f4e0d9e0f01655940f9220700484"
server_key_version: "encryptionkey@0"
I20260128 20:51:22.256865 11398 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:51:22.259657 11413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:22.260949 11398 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:22.261211 11398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "f8680c9dd37f419089d7388e87429646"
format_stamp: "Formatted at 2026-01-28 20:51:22 on dist-test-slave-hss7"
server_key: "9c4290b0beb248bafa1088ef93bb0875"
server_key_iv: "bd40f4e0d9e0f01655940f9220700484"
server_key_version: "encryptionkey@0"
I20260128 20:51:22.261349 11398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:22.273926 11398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:22.277472 11398 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:22.277702 11398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:22.278322 11398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:22.279515 11398 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:22.279585 11398 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:22.279663 11398 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:22.279716 11398 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:22.289664 11398 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:37543
I20260128 20:51:22.289731 11526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:37543 every 8 connection(s)
I20260128 20:51:22.290818 11398 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:22.297760 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11398
I20260128 20:51:22.297886 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:22.298149 4830 external_mini_cluster.cc:1468] Setting key b668ba9a94986290d03aa2c5b991225f
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:22.305553 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.292915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:39817 (local address 127.4.183.190:39147)
0128 20:51:22.293067 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.293071 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.293878 (+ 807us) server_negotiation.cc:408] Connection header received
0128 20:51:22.294847 (+ 969us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.294851 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.294917 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.295017 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.296862 (+ 1845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.297478 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.298369 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.298561 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.301597 (+ 3036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:22.301625 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:22.301628 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:22.301660 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:22.303230 (+ 1570us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.303903 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.303906 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.303907 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.303957 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.304315 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.304318 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.304319 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.304499 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:22.304582 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:22.305276 (+ 694us) server_negotiation.cc:300] Negotiation successful
0128 20:51:22.305412 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260128 20:51:22.306309 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.293038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39147 (local address 127.4.183.131:39817)
0128 20:51:22.293717 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:22.293760 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:22.294610 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:22.295194 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:22.295204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:22.295647 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:22.296667 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:22.296686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.297644 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.297647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:22.298158 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:22.298169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.298544 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.299256 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:22.299302 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:22.301416 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:22.303397 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:22.303403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:22.303415 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:22.303775 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:22.304095 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:22.304098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:22.304100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:22.304222 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:22.304638 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:22.304645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:22.304936 (+ 291us) client_negotiation.cc:769] Sending connection context
0128 20:51:22.305215 (+ 279us) client_negotiation.cc:240] Negotiation successful
0128 20:51:22.305500 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":250,"threads_started":1}
I20260128 20:51:22.307723 11527 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39147
I20260128 20:51:22.307983 11527 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:22.308660 11527 heartbeater.cc:507] Master 127.4.183.190:39147 requested a full tablet report, sending...
I20260128 20:51:22.310046 11062 ts_manager.cc:194] Registered new tserver with Master: f8680c9dd37f419089d7388e87429646 (127.4.183.131:37543)
I20260128 20:51:22.310755 11062 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:39817
I20260128 20:51:22.313089 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/test-admin.keytab.
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.332718 4830 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260128 20:51:22.336057 11534 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:22.334383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.4.183.129:39185)
0128 20:51:22.334819 (+ 436us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.334827 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.334854 (+ 27us) server_negotiation.cc:408] Connection header received
0128 20:51:22.334909 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.334914 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.335094 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.335260 (+ 166us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.335879 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55354: BlockingRecv error: recv got EOF from 127.0.0.1:55354 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":64,"threads_started":1}
W20260128 20:51:22.336549 4830 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc' of type FILE
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.338145 4830 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.339728 4830 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:51:22.352401 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.341096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.4.183.129:39185)
0128 20:51:22.341322 (+ 226us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.341327 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.341486 (+ 159us) server_negotiation.cc:408] Connection header received
0128 20:51:22.341564 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.341567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.341636 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.341724 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.342905 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.343676 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.344719 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.345039 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.347936 (+ 2897us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:22.348009 (+ 73us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:22.348027 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:22.348071 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:22.350461 (+ 2390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.351048 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.351055 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.351058 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.351134 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.351356 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.351359 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.351361 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.351617 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:22.351844 (+ 227us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:22.352043 (+ 199us) server_negotiation.cc:300] Negotiation successful
0128 20:51:22.352217 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260128 20:51:22.357671 11534 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:22.356139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.4.183.129:39185)
0128 20:51:22.356298 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.356302 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.356478 (+ 176us) server_negotiation.cc:408] Connection header received
0128 20:51:22.356568 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.356571 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.356658 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.356794 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.357523 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55382: BlockingRecv error: recv got EOF from 127.0.0.1:55382 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260128 20:51:22.357698 4830 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc' of type FILE
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.358982 4830 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.360119 4830 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:51:22.369964 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.361194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.4.183.129:39185)
0128 20:51:22.361331 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.361336 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.361422 (+ 86us) server_negotiation.cc:408] Connection header received
0128 20:51:22.361615 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.361618 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.361674 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.361769 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.362446 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.362938 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.363567 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.363781 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.366209 (+ 2428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:22.366236 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:22.366238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:22.366275 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:22.367915 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.368452 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.368455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.368456 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.368509 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.368825 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.368827 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.368829 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.369009 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:22.369119 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:22.369663 (+ 544us) server_negotiation.cc:300] Negotiation successful
0128 20:51:22.369816 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260128 20:51:22.373337 11534 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:22.372186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.4.183.129:39185)
0128 20:51:22.372344 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.372348 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.372371 (+ 23us) server_negotiation.cc:408] Connection header received
0128 20:51:22.372563 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.372566 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.372616 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.372701 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.373232 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55402: BlockingRecv error: recv got EOF from 127.0.0.1:55402 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260128 20:51:22.373448 4830 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769633420334841-4830-0/krb5kdc/krb5cc' of type FILE
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.374692 4830 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:22.375651 4830 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.129@KRBTEST.COM
I20260128 20:51:22.388825 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:22.377057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.4.183.129:39185)
0128 20:51:22.377228 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:22.377233 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:22.377410 (+ 177us) server_negotiation.cc:408] Connection header received
0128 20:51:22.377573 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:22.377577 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:22.377634 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:22.377724 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:22.378497 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.379209 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:22.381033 (+ 1824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:22.381326 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:22.384569 (+ 3243us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:22.384625 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:22.384628 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:22.384672 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:22.386539 (+ 1867us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.387089 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.387093 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.387094 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.387149 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:22.387489 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:22.387492 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:22.387494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:22.387695 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:22.387852 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:22.388308 (+ 456us) server_negotiation.cc:300] Negotiation successful
0128 20:51:22.388683 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260128 20:51:22.390159 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11126
I20260128 20:51:22.397435 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11262
I20260128 20:51:22.403707 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11398
I20260128 20:51:22.410601 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11030
2026-01-28T20:51:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3481 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11547](info): set up 2 sockets
Jan 28 20:51:22 dist-test-slave-hss7 krb5kdc[11547](info): commencing operation
krb5kdc: starting...
W20260128 20:51:24.483685 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.048s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:51:24 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:51:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:24.654778 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:45085
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:24.790764 11563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:24.791136 11563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:24.791253 11563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:24.795692 11563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:24.795866 11563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:24.795914 11563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:24.795958 11563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:24.796001 11563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:24.801182 11563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:45085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:24.802479 11563 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:24.803442 11563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:24.809924 11568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:24.809916 11569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:24.809924 11571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:24.810180 11563 server_base.cc:1060] running on GCE node
I20260128 20:51:24.810885 11563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:24.811951 11563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:24.813215 11563 hybrid_clock.cc:648] HybridClock initialized: now 1769633484813201 us; error 92 us; skew 500 ppm
Jan 28 20:51:24 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:24.816610 11563 init.cc:377] Logged in from keytab as oryx/127.4.183.190@KRBTEST.COM (short username oryx)
I20260128 20:51:24.818053 11563 webserver.cc:492] Webserver started at http://127.4.183.190:45695/ using document root <none> and password file <none>
I20260128 20:51:24.818691 11563 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:24.818775 11563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:24.819008 11563 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:24.821238 11563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "55b783e56be54266bb6628d8f392e2ef"
format_stamp: "Formatted at 2026-01-28 20:51:24 on dist-test-slave-hss7"
server_key: "cac084ba153cb01d6cfb8c63277f1876"
server_key_iv: "9dd79cf15ff625b4fb661c22dd830a8f"
server_key_version: "encryptionkey@0"
I20260128 20:51:24.821879 11563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "55b783e56be54266bb6628d8f392e2ef"
format_stamp: "Formatted at 2026-01-28 20:51:24 on dist-test-slave-hss7"
server_key: "cac084ba153cb01d6cfb8c63277f1876"
server_key_iv: "9dd79cf15ff625b4fb661c22dd830a8f"
server_key_version: "encryptionkey@0"
I20260128 20:51:24.826205 11563 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:51:24.829588 11578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:24.831393 11563 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260128 20:51:24.831645 11563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "55b783e56be54266bb6628d8f392e2ef"
format_stamp: "Formatted at 2026-01-28 20:51:24 on dist-test-slave-hss7"
server_key: "cac084ba153cb01d6cfb8c63277f1876"
server_key_iv: "9dd79cf15ff625b4fb661c22dd830a8f"
server_key_version: "encryptionkey@0"
I20260128 20:51:24.831799 11563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:24.846285 11563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:24.850502 11563 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:24.850754 11563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:24.859699 11630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:45085 every 8 connection(s)
I20260128 20:51:24.859695 11563 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:45085
I20260128 20:51:24.860965 11563 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:24.863683 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11563
I20260128 20:51:24.863933 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:24.864454 4830 external_mini_cluster.cc:1468] Setting key e0eaae903f169a3746d1a6490d55325c
I20260128 20:51:24.864946 11631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:24.872709 11631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Bootstrap starting.
Jan 28 20:51:24 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:24.875537 11631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:24.876591 11631 log.cc:826] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:24.879278 11631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: No bootstrap required, opened a new log
I20260128 20:51:24.881177 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:24.866086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.4.183.190:45085)
0128 20:51:24.866624 (+ 538us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:24.866638 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:24.866688 (+ 50us) server_negotiation.cc:408] Connection header received
0128 20:51:24.867973 (+ 1285us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:24.868006 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:24.868462 (+ 456us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:24.868905 (+ 443us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:24.870411 (+ 1506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:24.871587 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:24.872393 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:24.872682 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:24.875646 (+ 2964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:24.875670 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:24.875687 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:24.875752 (+ 65us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:24.878089 (+ 2337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:24.878678 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:24.878684 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:24.878690 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:24.878800 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:24.879093 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:24.879096 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:24.879098 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:24.879558 (+ 460us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:24.879761 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:24.880056 (+ 295us) server_negotiation.cc:300] Negotiation successful
0128 20:51:24.880334 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":166,"threads_started":1}
I20260128 20:51:24.882896 11631 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } }
I20260128 20:51:24.883140 11631 raft_consensus.cc:385] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:24.883178 11631 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b783e56be54266bb6628d8f392e2ef, State: Initialized, Role: FOLLOWER
I20260128 20:51:24.883699 11631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } }
I20260128 20:51:24.883846 11631 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:24.883950 11631 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:24.884081 11631 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:24.885277 11631 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } }
I20260128 20:51:24.885684 11631 leader_election.cc:304] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b783e56be54266bb6628d8f392e2ef; no voters:
I20260128 20:51:24.886027 11631 leader_election.cc:290] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:24.886257 11636 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:24.886611 11636 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 LEADER]: Becoming Leader. State: Replica: 55b783e56be54266bb6628d8f392e2ef, State: Running, Role: LEADER
I20260128 20:51:24.887426 11631 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:24.887316 11636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } }
I20260128 20:51:24.889918 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b783e56be54266bb6628d8f392e2ef. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } }
I20260128 20:51:24.890460 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:24.889935 11637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } }
I20260128 20:51:24.890573 11637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
W20260128 20:51:24.892900 11650 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:24.892998 11650 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:24.893076 11651 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:24.893993 11651 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:24.901036 11651 catalog_manager.cc:1357] Generated new cluster ID: 4fbd71821df24198bcc7e6c817ea9858
I20260128 20:51:24.901182 11651 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:24.913807 11651 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:24.915244 11651 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:24.926661 11651 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Generated new TSK 0
I20260128 20:51:24.927588 11651 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:25.001550 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:45085
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:25.154036 11659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:25.154415 11659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:25.154533 11659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:25.159576 11659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:25.159704 11659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:25.159921 11659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:25.166226 11659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:25.167555 11659 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:25.168541 11659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:25.177302 11665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:25.177331 11667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:25.177325 11664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:25.177675 11659 server_base.cc:1060] running on GCE node
I20260128 20:51:25.178128 11659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:25.178813 11659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:25.180074 11659 hybrid_clock.cc:648] HybridClock initialized: now 1769633485180010 us; error 65 us; skew 500 ppm
Jan 28 20:51:25 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633485, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:25.184427 11659 init.cc:377] Logged in from keytab as oryx/127.4.183.129@KRBTEST.COM (short username oryx)
I20260128 20:51:25.185986 11659 webserver.cc:492] Webserver started at http://127.4.183.129:33121/ using document root <none> and password file <none>
I20260128 20:51:25.186651 11659 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:25.186738 11659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:25.186975 11659 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:25.189859 11659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "353acb53f731489badbeb29923c2f9dc"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "51ad8665831f934fffa0fc67556d8376"
server_key_iv: "106607897d6924625b4c7ad898bf68be"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.190626 11659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "353acb53f731489badbeb29923c2f9dc"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "51ad8665831f934fffa0fc67556d8376"
server_key_iv: "106607897d6924625b4c7ad898bf68be"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.195199 11659 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:25.198163 11674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.199499 11659 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:25.199717 11659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "353acb53f731489badbeb29923c2f9dc"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "51ad8665831f934fffa0fc67556d8376"
server_key_iv: "106607897d6924625b4c7ad898bf68be"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.199854 11659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:25.211022 11659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:25.215554 11659 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:25.215862 11659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:25.216539 11659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:25.217681 11659 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:25.217765 11659 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.217849 11659 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:25.217893 11659 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.231530 11659 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:46141
I20260128 20:51:25.231547 11787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:46141 every 8 connection(s)
I20260128 20:51:25.232753 11659 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:25.241410 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11659
I20260128 20:51:25.241561 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:25.241873 4830 external_mini_cluster.cc:1468] Setting key 7b87ac4fa935b965d58ad64d7f47a95c
Jan 28 20:51:25 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633485, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:25.251552 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:25.235141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:55569 (local address 127.4.183.190:45085)
0128 20:51:25.235333 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:25.235339 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:25.236353 (+ 1014us) server_negotiation.cc:408] Connection header received
0128 20:51:25.237535 (+ 1182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:25.237539 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:25.237604 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:25.237836 (+ 232us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:25.240355 (+ 2519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.240948 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:25.242136 (+ 1188us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.242334 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:25.245471 (+ 3137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:25.245505 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:25.245510 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:25.245557 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:25.247535 (+ 1978us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:25.248753 (+ 1218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:25.248760 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:25.248763 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:25.248855 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:25.249445 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:25.249448 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:25.249450 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:25.249671 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:25.249775 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:25.251199 (+ 1424us) server_negotiation.cc:300] Negotiation successful
0128 20:51:25.251402 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260128 20:51:25.253016 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:25.235492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:55569)
0128 20:51:25.236160 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:25.236206 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:25.237102 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:25.238307 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:25.238319 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:25.238896 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:25.240066 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:25.240092 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.241233 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:25.241237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:25.241943 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:25.241961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.242294 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:25.243027 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:25.243069 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:25.245263 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:25.247813 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:25.247829 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:25.247854 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:25.248519 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:25.249011 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:25.249015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:25.249017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:25.249279 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:25.249833 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:25.249847 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:25.250427 (+ 580us) client_negotiation.cc:769] Sending connection context
0128 20:51:25.251062 (+ 635us) client_negotiation.cc:240] Negotiation successful
0128 20:51:25.251667 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":171,"threads_started":1}
WARNING: no policy specified for oryx/127.4.183.130@KRBTEST.COM; defaulting to no policy
I20260128 20:51:25.255543 11788 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45085
I20260128 20:51:25.255967 11788 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:25.256846 11788 heartbeater.cc:507] Master 127.4.183.190:45085 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.130@KRBTEST.COM" created.
I20260128 20:51:25.259276 11595 ts_manager.cc:194] Registered new tserver with Master: 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141)
I20260128 20:51:25.260656 11595 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:55569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:25.310340 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:45085
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:25.443459 11795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:25.443845 11795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:25.443948 11795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:25.447814 11795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:25.447959 11795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:25.448091 11795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:25.454566 11795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:25.455955 11795 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:25.456976 11795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:25.465378 11800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:25.465378 11801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:25.465412 11803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:25.465857 11795 server_base.cc:1060] running on GCE node
I20260128 20:51:25.466471 11795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:25.467175 11795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:25.468348 11795 hybrid_clock.cc:648] HybridClock initialized: now 1769633485468338 us; error 38 us; skew 500 ppm
Jan 28 20:51:25 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633485, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:25.471798 11795 init.cc:377] Logged in from keytab as oryx/127.4.183.130@KRBTEST.COM (short username oryx)
I20260128 20:51:25.473317 11795 webserver.cc:492] Webserver started at http://127.4.183.130:33685/ using document root <none> and password file <none>
I20260128 20:51:25.473959 11795 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:25.474020 11795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:25.474244 11795 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:25.476190 11795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "fcceb005c7c548679dd16848295150e7"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "e8e906c01dfa74f8b852427398b4819b"
server_key_iv: "7ca2845100b6bae07a0d73981c5cf27d"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.476735 11795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "fcceb005c7c548679dd16848295150e7"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "e8e906c01dfa74f8b852427398b4819b"
server_key_iv: "7ca2845100b6bae07a0d73981c5cf27d"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.480852 11795 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260128 20:51:25.484028 11810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.485455 11795 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:25.485649 11795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "fcceb005c7c548679dd16848295150e7"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "e8e906c01dfa74f8b852427398b4819b"
server_key_iv: "7ca2845100b6bae07a0d73981c5cf27d"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.485788 11795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:25.508625 11795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:25.512090 11795 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:25.512362 11795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:25.513025 11795 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:25.514148 11795 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:25.514235 11795 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.514309 11795 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:25.514358 11795 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.526165 11795 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:45699
I20260128 20:51:25.526180 11923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:45699 every 8 connection(s)
I20260128 20:51:25.527433 11795 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:25.529803 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11795
I20260128 20:51:25.529928 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:25.530215 4830 external_mini_cluster.cc:1468] Setting key c2c32cea37d05ed292786859b29eabb1
Jan 28 20:51:25 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633485, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.131@KRBTEST.COM" created.
I20260128 20:51:25.546386 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:25.529831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:57417 (local address 127.4.183.190:45085)
0128 20:51:25.530004 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:25.530010 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:25.531366 (+ 1356us) server_negotiation.cc:408] Connection header received
0128 20:51:25.532731 (+ 1365us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:25.532735 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:25.532802 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:25.532906 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:25.534991 (+ 2085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.535633 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:25.536347 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.536537 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:25.539961 (+ 3424us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:25.540002 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:25.540005 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:25.540063 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:25.542501 (+ 2438us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:25.544237 (+ 1736us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:25.544242 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:25.544244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:25.544363 (+ 119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:25.544850 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:25.544854 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:25.544856 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:25.545061 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:25.545247 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:25.546007 (+ 760us) server_negotiation.cc:300] Negotiation successful
0128 20:51:25.546196 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260128 20:51:25.547302 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:25.530161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:57417)
0128 20:51:25.531127 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:25.531196 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:25.532460 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:25.533079 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:25.533089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:25.533642 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:25.534766 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:25.534789 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.535796 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:25.535800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:25.536213 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:25.536223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.536515 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:25.537322 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:25.537352 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:25.539566 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:25.542891 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:25.542911 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:25.542931 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:25.543937 (+ 1006us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:25.544565 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:25.544568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:25.544570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:25.544722 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:25.545215 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:25.545224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:25.545549 (+ 325us) client_negotiation.cc:769] Sending connection context
0128 20:51:25.545963 (+ 414us) client_negotiation.cc:240] Negotiation successful
0128 20:51:25.546304 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":185,"threads_started":1}
I20260128 20:51:25.549005 11924 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45085
I20260128 20:51:25.549374 11924 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:25.550056 11924 heartbeater.cc:507] Master 127.4.183.190:45085 requested a full tablet report, sending...
I20260128 20:51:25.551455 11595 ts_manager.cc:194] Registered new tserver with Master: fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:25.552213 11595 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:57417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:25.593789 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:45085
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:25.725108 11931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:25.725430 11931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:25.725493 11931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:25.729056 11931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:25.729190 11931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:25.729278 11931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:25.734253 11931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:25.735432 11931 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:25.736330 11931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:25.745781 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:25.745796 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:25.745868 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:25.746429 11931 server_base.cc:1060] running on GCE node
I20260128 20:51:25.747080 11931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:25.748064 11931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:25.749225 11931 hybrid_clock.cc:648] HybridClock initialized: now 1769633485749209 us; error 46 us; skew 500 ppm
Jan 28 20:51:25 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633485, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:25.752996 11931 init.cc:377] Logged in from keytab as oryx/127.4.183.131@KRBTEST.COM (short username oryx)
I20260128 20:51:25.754503 11931 webserver.cc:492] Webserver started at http://127.4.183.131:41199/ using document root <none> and password file <none>
I20260128 20:51:25.755209 11931 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:25.755308 11931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:25.755560 11931 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:25.757655 11931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "27e7ed4ed52848ce89ffdc349ae45ca5"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "cbeef5355f6491dc07091c051b2a72db"
server_key_iv: "667b3e8191f4433710759732260bb8a8"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.758265 11931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "27e7ed4ed52848ce89ffdc349ae45ca5"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "cbeef5355f6491dc07091c051b2a72db"
server_key_iv: "667b3e8191f4433710759732260bb8a8"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.762806 11931 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260128 20:51:25.766741 11946 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.768219 11931 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:25.768442 11931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "27e7ed4ed52848ce89ffdc349ae45ca5"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "cbeef5355f6491dc07091c051b2a72db"
server_key_iv: "667b3e8191f4433710759732260bb8a8"
server_key_version: "encryptionkey@0"
I20260128 20:51:25.768589 11931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:25.789705 11931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:25.793468 11931 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:25.793715 11931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:25.794373 11931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:25.795481 11931 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:25.795581 11931 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.795660 11931 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:25.795704 11931 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:25.807607 11931 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:40173
I20260128 20:51:25.807623 12059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:40173 every 8 connection(s)
I20260128 20:51:25.808816 11931 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:25.813185 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 11931
I20260128 20:51:25.813349 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:25.813657 4830 external_mini_cluster.cc:1468] Setting key e1c4df1f754ebbf62d23362f310058f1
Jan 28 20:51:25 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633485, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:25.825896 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:25.811211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:51871 (local address 127.4.183.190:45085)
0128 20:51:25.811375 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:25.811378 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:25.812228 (+ 850us) server_negotiation.cc:408] Connection header received
0128 20:51:25.813407 (+ 1179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:25.813411 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:25.813490 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:25.813580 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:25.816135 (+ 2555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.816688 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:25.817492 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.817683 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:25.820526 (+ 2843us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:25.820554 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:25.820557 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:25.820594 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:25.822546 (+ 1952us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:25.823247 (+ 701us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:25.823253 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:25.823255 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:25.823324 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:25.823665 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:25.823669 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:25.823671 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:25.824118 (+ 447us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:25.824354 (+ 236us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:25.825234 (+ 880us) server_negotiation.cc:300] Negotiation successful
0128 20:51:25.825656 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260128 20:51:25.826763 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:25.811473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:51871)
0128 20:51:25.812038 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:25.812082 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:25.813028 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:25.813875 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:25.813889 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:25.814535 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:25.815948 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:25.815973 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.816836 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:25.816841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:25.817345 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:25.817368 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:25.817657 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:25.818358 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:25.818393 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:25.820275 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:25.822782 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:25.822790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:25.822804 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:25.823117 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:25.823441 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:25.823445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:25.823447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:25.823563 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:25.824376 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:25.824391 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:25.824821 (+ 430us) client_negotiation.cc:769] Sending connection context
0128 20:51:25.825138 (+ 317us) client_negotiation.cc:240] Negotiation successful
0128 20:51:25.825770 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":145,"threads_started":1}
I20260128 20:51:25.828070 12060 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45085
I20260128 20:51:25.828397 12060 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:25.829015 12060 heartbeater.cc:507] Master 127.4.183.190:45085 requested a full tablet report, sending...
I20260128 20:51:25.830426 11595 ts_manager.cc:194] Registered new tserver with Master: 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173)
I20260128 20:51:25.831131 11595 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:51871
I20260128 20:51:25.842406 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.183.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:25.905432 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.189:45875
--webserver_interface=127.4.183.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:45085,127.4.183.189:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:26.029915 12067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:26.030248 12067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:26.030316 12067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:26.034020 12067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:26.034132 12067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:26.034159 12067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:26.034183 12067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:26.034204 12067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:26.039404 12067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:45085,127.4.183.189:45875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.189:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.183.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
W20260128 20:51:26.039710 12067 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.183.190:45085,127.4.183.189:45875'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260128 20:51:26.041200 12067 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:26.043013 12067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:26.050441 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.050459 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.050441 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:26.051121 12067 server_base.cc:1060] running on GCE node
I20260128 20:51:26.051728 12067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:26.053002 12067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:26.054260 12067 hybrid_clock.cc:648] HybridClock initialized: now 1769633486054230 us; error 59 us; skew 500 ppm
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:26.058744 12067 init.cc:377] Logged in from keytab as oryx/127.4.183.189@KRBTEST.COM (short username oryx)
I20260128 20:51:26.060822 12067 webserver.cc:492] Webserver started at http://127.4.183.189:33327/ using document root <none> and password file <none>
I20260128 20:51:26.061681 12067 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:26.061789 12067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:26.062073 12067 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:26.064254 12067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/instance:
uuid: "28230a487eeb47ca815837c6f045f379"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "0cfbf6fc8ca375ef0ba90d3baa971d09"
server_key_iv: "c36936314ee747fe0273cf8c1341ff0f"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.064931 12067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal/instance:
uuid: "28230a487eeb47ca815837c6f045f379"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "0cfbf6fc8ca375ef0ba90d3baa971d09"
server_key_iv: "c36936314ee747fe0273cf8c1341ff0f"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.069403 12067 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:51:26.072722 12082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:26.074353 12067 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:26.074559 12067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
uuid: "28230a487eeb47ca815837c6f045f379"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "0cfbf6fc8ca375ef0ba90d3baa971d09"
server_key_iv: "c36936314ee747fe0273cf8c1341ff0f"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.074698 12067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:26.105839 12067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:26.109472 12067 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:26.109725 12067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:26.118790 12067 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.189:45875
I20260128 20:51:26.118784 12134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.189:45875 every 8 connection(s)
I20260128 20:51:26.120000 12067 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/info.pb
I20260128 20:51:26.123487 12135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:26.124514 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 12067
I20260128 20:51:26.129281 12135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.189@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:26.141564 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.126093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.4.183.189:45875)
0128 20:51:26.126582 (+ 489us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.126592 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.126640 (+ 48us) server_negotiation.cc:408] Connection header received
0128 20:51:26.127571 (+ 931us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.127598 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.127969 (+ 371us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.128300 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.129566 (+ 1266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.130438 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.131330 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.131659 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.134764 (+ 3105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.134827 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.134839 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.134893 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.137295 (+ 2402us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.138052 (+ 757us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.138062 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.138071 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.138225 (+ 154us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.138943 (+ 718us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.138950 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.138957 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.139463 (+ 506us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.139750 (+ 287us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.140137 (+ 387us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.140524 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":182,"threads_started":1}
I20260128 20:51:26.141927 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.131287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56798)
0128 20:51:26.132134 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.132171 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.132289 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.132673 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.132679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.133095 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.133442 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.133450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.134556 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.134561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.135066 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.135075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.135176 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.135792 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.135806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.137679 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.139884 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.139894 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.139900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.140277 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.140684 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.140687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.140688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.140742 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.141263 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.141270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.141383 (+ 113us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.141518 (+ 135us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.141738 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":166,"threads_started":1}
I20260128 20:51:26.141934 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.131374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.4.183.190:45085)
0128 20:51:26.131540 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.131544 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.132274 (+ 730us) server_negotiation.cc:408] Connection header received
0128 20:51:26.132394 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.132401 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.132477 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.132582 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.133594 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.134248 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.135215 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.135395 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.137836 (+ 2441us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.137866 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.137873 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.137916 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.139695 (+ 1779us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.140471 (+ 776us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.140477 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.140480 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.140564 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.140842 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.140846 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.140848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.141104 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.141262 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.141620 (+ 358us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.141769 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260128 20:51:26.143098 12135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.189@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
I20260128 20:51:26.151790 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.143746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39270)
0128 20:51:26.143919 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.143936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.144013 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.144226 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.144229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.144388 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.144633 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.144641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.145408 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.145413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.145767 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.145774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.145904 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.146502 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.146516 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.147954 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.150245 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.150249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.150251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.150444 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.150710 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.150715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.150717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.150766 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.151153 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.151156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.151233 (+ 77us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.151338 (+ 105us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.151535 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260128 20:51:26.151790 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.143915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.4.183.189:45875)
0128 20:51:26.144033 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.144040 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.144058 (+ 18us) server_negotiation.cc:408] Connection header received
0128 20:51:26.144090 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.144093 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.144131 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.144204 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.144753 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.145301 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.145879 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.146070 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.148105 (+ 2035us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.148140 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.148142 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.148164 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.150063 (+ 1899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.150555 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.150559 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.150561 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.150616 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.150859 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.150865 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.150867 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.151050 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.151155 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.151392 (+ 237us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.151619 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":44}
I20260128 20:51:26.155771 12135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Bootstrap starting.
I20260128 20:51:26.158634 12135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:26.159466 12135 log.cc:826] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:26.162396 12135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: No bootstrap required, opened a new log
I20260128 20:51:26.166885 12135 raft_consensus.cc:359] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } }
I20260128 20:51:26.167208 12135 raft_consensus.cc:385] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:26.167304 12135 raft_consensus.cc:740] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28230a487eeb47ca815837c6f045f379, State: Initialized, Role: FOLLOWER
I20260128 20:51:26.167870 12135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } }
I20260128 20:51:26.168499 12141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } }
I20260128 20:51:26.168649 12141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:26.169374 12135 sys_catalog.cc:565] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: configured and running, proceeding with master startup.
W20260128 20:51:26.175477 12152 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:26.175616 12152 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:26.178460 12067 master_runner.cc:428] Detected that this master 28230a487eeb47ca815837c6f045f379 is joining an existing cluster
I20260128 20:51:26.178541 12067 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.183.189:45875
I20260128 20:51:26.190536 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.181466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.4.183.189:45875)
0128 20:51:26.181553 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.181558 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.181685 (+ 127us) server_negotiation.cc:408] Connection header received
0128 20:51:26.181871 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.181875 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.181916 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.181987 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.183446 (+ 1459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.184206 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.185726 (+ 1520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.185926 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.186674 (+ 748us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.186692 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.186696 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.186724 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.188595 (+ 1871us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.189129 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.189171 (+ 42us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.189173 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.189231 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.189484 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.189487 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.189489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.189708 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.189817 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.190223 (+ 406us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.190355 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260128 20:51:26.190575 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.181004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39274)
0128 20:51:26.181611 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.181624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.181731 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.182097 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.182100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.182299 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.183214 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.183229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.184535 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.184541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.185582 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.185594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.185726 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.186228 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.186242 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.186530 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.188825 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.188829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.188831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.189017 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.189339 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.189341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.189343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.189391 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.189891 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.189895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.190039 (+ 144us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.190146 (+ 107us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.190278 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":155,"threads_started":1}
WARNING: no policy specified for oryx/127.4.183.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.188@KRBTEST.COM" created.
I20260128 20:51:26.210490 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.180974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56806)
0128 20:51:26.181500 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.181526 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.181657 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.182019 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.182023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.182299 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.182681 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.182691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.183959 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.183962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.184732 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.184742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.184903 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.185590 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.185609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.205588 (+ 19979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.208003 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.208012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.208017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.208296 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.209235 (+ 939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.209239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.209241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.209315 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.209845 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.209848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.209973 (+ 125us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.210098 (+ 125us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.210278 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"spinlock_wait_cycles":2944,"thread_start_us":161,"threads_started":1}
I20260128 20:51:26.210585 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.181052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.4.183.190:45085)
0128 20:51:26.181239 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.181244 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.181603 (+ 359us) server_negotiation.cc:408] Connection header received
0128 20:51:26.181802 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.181805 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.181889 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.181982 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.182942 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.183768 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.184920 (+ 1152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.185207 (+ 287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.205857 (+ 20650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.205890 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.205894 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.205939 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.207741 (+ 1802us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.208416 (+ 675us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.208423 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.208427 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.208490 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.209446 (+ 956us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.209453 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.209457 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.209725 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.209849 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.210188 (+ 339us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.210351 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.190@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:26.230743 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.215866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.4.183.189:45875)
0128 20:51:26.216004 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.216009 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.216675 (+ 666us) server_negotiation.cc:408] Connection header received
0128 20:51:26.216920 (+ 245us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.216923 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.216975 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.217067 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.218330 (+ 1263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.219012 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.220190 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.220426 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.223886 (+ 3460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.223928 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.223931 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.223970 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.228654 (+ 4684us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.229389 (+ 735us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.229393 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.229395 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.229449 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.229719 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.229722 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.229724 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.229939 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.230053 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.230417 (+ 364us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.230560 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 20:51:26.230748 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.215827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39286)
0128 20:51:26.216553 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.216585 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.216769 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.217106 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.217111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.217612 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.218070 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.218095 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.219371 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.219375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.220043 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.220055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.220232 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.221225 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.221247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.223646 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.228858 (+ 5212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.228881 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.228888 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.229274 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.229557 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.229561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.229563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.229622 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.230058 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.230069 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.230204 (+ 135us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.230354 (+ 150us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.230563 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":108,"threads_started":1}
I20260128 20:51:26.233107 11595 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.4.183.189:45875: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "28230a487eeb47ca815837c6f045f379"
member_type: NON_VOTER
last_known_addr {
host: "127.4.183.189"
port: 45875
}
attrs {
promote: true
}
}
dest_uuid: "55b783e56be54266bb6628d8f392e2ef"
cas_config_opid_index: -1
I20260128 20:51:26.234129 11595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } }
I20260128 20:51:26.239339 12166 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 28230a487eeb47ca815837c6f045f379 (127.4.183.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.240572 12109 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:26.241386 12166 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.4.183.189:45875
I20260128 20:51:26.242004 12067 master.cc:561] Master@127.4.183.189:45875 shutting down...
I20260128 20:51:26.242537 12109 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Refusing update from remote peer 55b783e56be54266bb6628d8f392e2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
WARNING: no policy specified for HTTP/127.4.183.188@KRBTEST.COM; defaulting to no policy
I20260128 20:51:26.244232 12167 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:26.245280 12165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.245409 12165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.188@KRBTEST.COM" created.
I20260128 20:51:26.248080 12067 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:26.248740 12067 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:26.248970 12067 tablet_replica.cc:333] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: stopping tablet replica
W20260128 20:51:26.249365 11579 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260128 20:51:26.250679 11579 consensus_peers.cc:597] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef -> Peer 28230a487eeb47ca815837c6f045f379 (127.4.183.189:45875): Couldn't send request to peer 28230a487eeb47ca815837c6f045f379. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260128 20:51:26.255869 12067 master.cc:583] Master@127.4.183.189:45875 shutdown complete.
I20260128 20:51:26.255959 12067 master_runner.cc:305] Clearing existing system tablet
I20260128 20:51:26.256223 12067 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:26.256291 12067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:26.258245 12067 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260128 20:51:26.259800 12173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:26.260082 12067 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:26.260236 12067 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260128 20:51:26.260296 12067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
uuid: "28230a487eeb47ca815837c6f045f379"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "0cfbf6fc8ca375ef0ba90d3baa971d09"
server_key_iv: "c36936314ee747fe0273cf8c1341ff0f"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.262477 12067 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:26.264979 11788 heartbeater.cc:499] Master 127.4.183.190:45085 was elected leader, sending a full tablet report...
I20260128 20:51:26.266176 12067 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260128 20:51:26.266325 12067 log.cc:1199] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260128 20:51:26.266634 12067 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Deleting consensus metadata
I20260128 20:51:26.266816 12067 master_runner.cc:315] Copying system tablet from 127.4.183.190:45085
I20260128 20:51:26.268507 12067 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: tablet copy: Beginning tablet copy session from remote peer at address 127.4.183.190:45085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:26.270282 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.188:42553
--webserver_interface=127.4.183.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 20:51:26.280272 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.269220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56816)
0128 20:51:26.269526 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.269549 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.269690 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.270111 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.270114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.270533 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.270881 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.270894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.272104 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.272111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.272789 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.272803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.272952 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.273691 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.273714 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.274217 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.277779 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.277789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.277794 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.278167 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.278559 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.278566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.278570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.278694 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.279323 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.279329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.279500 (+ 171us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.279671 (+ 171us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.279927 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260128 20:51:26.280375 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.269643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.4.183.190:45085)
0128 20:51:26.269784 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.269789 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.269805 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:51:26.269856 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.269860 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.269925 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.270017 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.271048 (+ 1031us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.271945 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.272946 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.273628 (+ 682us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.274384 (+ 756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.274411 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.274415 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.274446 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.277501 (+ 3055us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.278347 (+ 846us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.278352 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.278354 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.278430 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.278822 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.278826 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.278828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.279195 (+ 367us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.279313 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.279953 (+ 640us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.280176 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 20:51:26.281834 11615 tablet_copy_service.cc:140] P 55b783e56be54266bb6628d8f392e2ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 28230a487eeb47ca815837c6f045f379 ({username='oryx', principal='oryx/127.4.183.189@KRBTEST.COM'} at 127.0.0.1:56816)
I20260128 20:51:26.282112 11615 tablet_copy_service.cc:161] P 55b783e56be54266bb6628d8f392e2ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 28230a487eeb47ca815837c6f045f379 at {username='oryx', principal='oryx/127.4.183.189@KRBTEST.COM'} at 127.0.0.1:56816: session id = 28230a487eeb47ca815837c6f045f379-00000000000000000000000000000000
I20260128 20:51:26.284420 11615 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260128 20:51:26.287631 12067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260128 20:51:26.290867 12067 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: tablet copy: Starting download of 0 data blocks...
I20260128 20:51:26.291118 12067 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: tablet copy: Starting download of 1 WAL segments...
I20260128 20:51:26.295221 12067 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260128 20:51:26.297930 11615 tablet_copy_service.cc:342] P 55b783e56be54266bb6628d8f392e2ef: Request end of tablet copy session 28230a487eeb47ca815837c6f045f379-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.183.189@KRBTEST.COM'} at 127.0.0.1:56816
I20260128 20:51:26.298116 11615 tablet_copy_service.cc:434] P 55b783e56be54266bb6628d8f392e2ef: ending tablet copy session 28230a487eeb47ca815837c6f045f379-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 28230a487eeb47ca815837c6f045f379
W20260128 20:51:26.303730 12067 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260128 20:51:26.307618 12067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:26.310328 12181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.310472 12182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.310600 12184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:26.311290 12067 server_base.cc:1060] running on GCE node
I20260128 20:51:26.311496 12067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:26.311830 12067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:26.312978 12067 hybrid_clock.cc:648] HybridClock initialized: now 1769633486312954 us; error 43 us; skew 500 ppm
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:26.314807 12067 init.cc:377] Logged in from keytab as oryx/127.4.183.189@KRBTEST.COM (short username oryx)
I20260128 20:51:26.315374 12067 webserver.cc:492] Webserver started at http://127.4.183.189:43311/ using document root <none> and password file <none>
I20260128 20:51:26.315759 12067 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:26.315809 12067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:26.317569 12067 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:26.318717 12191 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:26.319146 12067 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:26.319316 12067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
uuid: "28230a487eeb47ca815837c6f045f379"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "0cfbf6fc8ca375ef0ba90d3baa971d09"
server_key_iv: "c36936314ee747fe0273cf8c1341ff0f"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.319576 12067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:26.339196 12067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:26.344853 12067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:26.350488 12067 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.189:45875
I20260128 20:51:26.350523 12243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.189:45875 every 8 connection(s)
I20260128 20:51:26.351244 12067 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-1/data/info.pb
I20260128 20:51:26.353286 12244 sys_catalog.cc:263] Verifying existing consensus state
I20260128 20:51:26.354526 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Bootstrap starting.
I20260128 20:51:26.367751 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260128 20:51:26.368157 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: Bootstrap complete.
I20260128 20:51:26.368930 12244 raft_consensus.cc:359] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } }
I20260128 20:51:26.369091 12244 raft_consensus.cc:740] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 28230a487eeb47ca815837c6f045f379, State: Initialized, Role: LEARNER
I20260128 20:51:26.369236 12244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } }
I20260128 20:51:26.369669 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.369803 12244 sys_catalog.cc:565] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:26.369897 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: LEARNER
I20260128 20:51:26.374300 12258 catalog_manager.cc:1269] Loaded cluster ID: 4fbd71821df24198bcc7e6c817ea9858
I20260128 20:51:26.374511 12258 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: loading cluster ID for follower catalog manager: success
I20260128 20:51:26.377841 12258 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: acquiring CA information for follower catalog manager: success
I20260128 20:51:26.379983 12258 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260128 20:51:26.433768 12180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:26.434083 12180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:26.434150 12180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:26.438197 12180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:26.438289 12180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:26.438314 12180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:26.438333 12180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:26.438385 12180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:26.443629 12180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.188:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.183.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:26.445117 12180 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:26.446228 12180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:26.452939 12264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.452926 12266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.452926 12263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:26.453243 12180 server_base.cc:1060] running on GCE node
I20260128 20:51:26.453820 12180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:26.454893 12180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:26.456112 12180 hybrid_clock.cc:648] HybridClock initialized: now 1769633486456082 us; error 53 us; skew 500 ppm
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:26.459566 12180 init.cc:377] Logged in from keytab as oryx/127.4.183.188@KRBTEST.COM (short username oryx)
I20260128 20:51:26.461553 12180 webserver.cc:492] Webserver started at http://127.4.183.188:38413/ using document root <none> and password file <none>
I20260128 20:51:26.462188 12180 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:26.462247 12180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:26.462435 12180 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:26.464337 12180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/instance:
uuid: "369959360abb4f4e89afcd777b02bbf1"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "f3c25968a48dbded8b21f61a5c709e27"
server_key_iv: "5cbee00bb638bdec460568ae82a81fcc"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.464861 12180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal/instance:
uuid: "369959360abb4f4e89afcd777b02bbf1"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "f3c25968a48dbded8b21f61a5c709e27"
server_key_iv: "5cbee00bb638bdec460568ae82a81fcc"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.469424 12180 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260128 20:51:26.473204 12273 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:26.474637 12180 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:26.474999 12180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
uuid: "369959360abb4f4e89afcd777b02bbf1"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "f3c25968a48dbded8b21f61a5c709e27"
server_key_iv: "5cbee00bb638bdec460568ae82a81fcc"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.475220 12180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:26.492426 12180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:26.495956 12180 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:26.496225 12180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:26.504823 12180 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.188:42553
I20260128 20:51:26.504823 12325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.188:42553 every 8 connection(s)
I20260128 20:51:26.506345 12180 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/info.pb
I20260128 20:51:26.509899 12326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:26.510584 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 12180
I20260128 20:51:26.515548 12326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.183.188@KRBTEST.COM
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.188@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:26.528959 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.512363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.4.183.188:42553)
0128 20:51:26.513272 (+ 909us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.513283 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.513325 (+ 42us) server_negotiation.cc:408] Connection header received
0128 20:51:26.514001 (+ 676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.514021 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.514363 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.514777 (+ 414us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.515827 (+ 1050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.516696 (+ 869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.517544 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.517883 (+ 339us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.521474 (+ 3591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.521536 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.521546 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.521597 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.524698 (+ 3101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.525485 (+ 787us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.525491 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.525497 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.525614 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.525962 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.525969 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.525974 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.526466 (+ 492us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.526758 (+ 292us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.527319 (+ 561us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.528019 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":178,"threads_started":1}
I20260128 20:51:26.530776 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.517787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.4.183.190:45085)
0128 20:51:26.517980 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.517985 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.518656 (+ 671us) server_negotiation.cc:408] Connection header received
0128 20:51:26.518857 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.518861 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.518921 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.519015 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.520557 (+ 1542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.521310 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.522067 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.522278 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.526022 (+ 3744us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.526064 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.526068 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.526127 (+ 59us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.528469 (+ 2342us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.529306 (+ 837us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.529311 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.529313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.529378 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.529692 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.529697 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.529699 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.529933 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.530058 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.530457 (+ 399us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.530619 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260128 20:51:26.531358 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.517622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56832)
0128 20:51:26.518488 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.518528 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.518732 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.519099 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.519107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.519633 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.520179 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.520207 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.521462 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.521469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.521904 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.521915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.522084 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.522912 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.522935 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.525661 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.528657 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.528667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.528672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.529008 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.529510 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.529513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.529515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.529574 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.530074 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.530081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.530207 (+ 126us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.530962 (+ 755us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.531171 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":548,"threads_started":1}
I20260128 20:51:26.532907 12326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.188@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
I20260128 20:51:26.542194 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.533529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39292)
0128 20:51:26.533705 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.533719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.533818 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.534244 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.534247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.534430 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.534629 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.534635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.535639 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.535643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.536081 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.536090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.536197 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.536835 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.536859 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.538298 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.540374 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.540378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.540379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.540578 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.540955 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.540957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.540959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.541051 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.541728 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.541731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.541825 (+ 94us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.541923 (+ 98us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.542055 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260128 20:51:26.542331 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.533757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.4.183.189:45875)
0128 20:51:26.534011 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.534014 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.534027 (+ 13us) server_negotiation.cc:408] Connection header received
0128 20:51:26.534065 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.534067 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.534110 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.534192 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.534789 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.535502 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.536216 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.536382 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.538475 (+ 2093us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.538501 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.538503 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.538536 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.540195 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.540721 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.540725 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.540726 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.540784 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.541309 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.541323 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.541328 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.541601 (+ 273us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.541697 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.542017 (+ 320us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.542143 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260128 20:51:26.543293 12326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.188@KRBTEST.COM for oryx/127.4.183.188@KRBTEST.COM
I20260128 20:51:26.552899 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.544301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.4.183.188:42553)
0128 20:51:26.544456 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.544462 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.544587 (+ 125us) server_negotiation.cc:408] Connection header received
0128 20:51:26.544743 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.544764 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.544808 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.544928 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.545488 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.546151 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.546765 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.546963 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.549405 (+ 2442us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.549434 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.549436 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.549460 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.551060 (+ 1600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.551527 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.551531 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.551533 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.551579 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.551962 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.551968 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.551970 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.552158 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.552273 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.552508 (+ 235us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.552723 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":61}
I20260128 20:51:26.552899 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.544262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47146)
0128 20:51:26.544456 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.544474 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.544651 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.544941 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.544948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.545201 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.545375 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.545382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.546300 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.546303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.546650 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.546657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.546795 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.547454 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.547613 (+ 159us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.549254 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.551251 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.551255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.551257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.551426 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.551679 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.551683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.551686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.551835 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.552272 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.552275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.552353 (+ 78us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.552465 (+ 112us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.552658 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260128 20:51:26.555076 11924 heartbeater.cc:499] Master 127.4.183.190:45085 was elected leader, sending a full tablet report...
I20260128 20:51:26.557204 12326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Bootstrap starting.
I20260128 20:51:26.560055 12326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:26.560920 12326 log.cc:826] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:26.563220 12326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: No bootstrap required, opened a new log
I20260128 20:51:26.566088 12326 raft_consensus.cc:359] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } }
I20260128 20:51:26.566303 12326 raft_consensus.cc:385] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:26.566340 12326 raft_consensus.cc:740] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369959360abb4f4e89afcd777b02bbf1, State: Initialized, Role: FOLLOWER
I20260128 20:51:26.566856 12326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } }
I20260128 20:51:26.567472 12332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } } }
I20260128 20:51:26.567631 12332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:26.568164 12326 sys_catalog.cc:565] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: configured and running, proceeding with master startup.
W20260128 20:51:26.574344 12343 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:26.574496 12343 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:26.580415 12180 master_runner.cc:428] Detected that this master 369959360abb4f4e89afcd777b02bbf1 is joining an existing cluster
I20260128 20:51:26.580513 12180 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.183.188:42553
I20260128 20:51:26.587560 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.584591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56858)
0128 20:51:26.584716 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.584732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.584819 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.585311 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.585314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.585328 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:26.585591 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.585612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.586402 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.586408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.587029 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.587039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.587194 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.587268 (+ 74us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.587327 (+ 59us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.587405 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260128 20:51:26.587949 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.584717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.4.183.190:45085)
0128 20:51:26.585023 (+ 306us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.585026 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.585036 (+ 10us) server_negotiation.cc:408] Connection header received
0128 20:51:26.585105 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.585111 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.585189 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.585697 (+ 508us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:26.585770 (+ 73us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.586286 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.587169 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.587598 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.587689 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.587741 (+ 52us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.587819 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":139,"threads_started":1}
I20260128 20:51:26.592317 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.589429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39308)
0128 20:51:26.589547 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.589569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.589741 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.590145 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.590148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.590166 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:26.590359 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.590366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.591154 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.591158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.591842 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.591853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.592003 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.592031 (+ 28us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.592088 (+ 57us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.592162 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260128 20:51:26.592976 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.589563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.4.183.189:45875)
0128 20:51:26.589893 (+ 330us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.589897 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.589908 (+ 11us) server_negotiation.cc:408] Connection header received
0128 20:51:26.589952 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.589955 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.590007 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.590135 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:26.590477 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.591026 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.591983 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.592614 (+ 631us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.592748 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.592799 (+ 51us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.592873 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
I20260128 20:51:26.610351 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.584514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.4.183.188:42553)
0128 20:51:26.584626 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.584631 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.585423 (+ 792us) server_negotiation.cc:408] Connection header received
0128 20:51:26.585557 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.585561 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.585597 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.585641 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.602257 (+ 16616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.603013 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.603831 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.604005 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.605551 (+ 1546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.605579 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.605582 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.605606 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.607923 (+ 2317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.608582 (+ 659us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.608586 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.608588 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.608640 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.608924 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.608928 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.608930 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.609254 (+ 324us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.609357 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.609750 (+ 393us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.610155 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260128 20:51:26.611042 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.584699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47156)
0128 20:51:26.585361 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.585373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.585470 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.585758 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.585761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.601641 (+ 15880us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.602047 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.602062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.603152 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.603155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.603714 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.603723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.604142 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.604827 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.604854 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.605412 (+ 558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.608262 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.608267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.608269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.608443 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.608740 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.608746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.608750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.608800 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.609405 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.609413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.609554 (+ 141us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.609911 (+ 357us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.610279 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":207,"spinlock_wait_cycles":9344,"thread_start_us":80,"threads_started":1}
I20260128 20:51:26.611893 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.583380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39300)
0128 20:51:26.583848 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.583863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.584012 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.584403 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.584409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.601614 (+ 17205us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.602046 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.602061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.603272 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.603276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.603919 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.603929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.604062 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.606305 (+ 2243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.606324 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.606787 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.609725 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.609729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.609732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.610023 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.610865 (+ 842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.610871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.610873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.610927 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.611365 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.611368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.611468 (+ 100us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.611578 (+ 110us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.611725 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":152,"threads_started":1}
I20260128 20:51:26.612917 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.583483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.4.183.189:45875)
0128 20:51:26.583655 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.583661 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.583932 (+ 271us) server_negotiation.cc:408] Connection header received
0128 20:51:26.584130 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.584133 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.584189 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.584243 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.602280 (+ 18037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.603109 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.604379 (+ 1270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.604609 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.606958 (+ 2349us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.606981 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.606985 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.607020 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.609568 (+ 2548us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.610658 (+ 1090us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.610663 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.610665 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.610731 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.611042 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.611046 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.611049 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.611255 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.611378 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.612131 (+ 753us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.612277 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260128 20:51:26.615078 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.583992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.4.183.190:45085)
0128 20:51:26.584391 (+ 399us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.584396 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.585116 (+ 720us) server_negotiation.cc:408] Connection header received
0128 20:51:26.585265 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.585270 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.585314 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.585381 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.602718 (+ 17337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.603532 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.605905 (+ 2373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.606123 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.608318 (+ 2195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.608338 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.608343 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.608376 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.610117 (+ 1741us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.610691 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.610695 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.610697 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.610751 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.611085 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.611090 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.611091 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.612432 (+ 1341us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.612575 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.614677 (+ 2102us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.614958 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260128 20:51:26.615412 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.584114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56846)
0128 20:51:26.585012 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.585025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.585169 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.585481 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.585484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.601640 (+ 16156us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.602523 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.602535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.604845 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.604850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.605515 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.605526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.605658 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.607342 (+ 1684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.607360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.608160 (+ 800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.610279 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.610283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.610284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.610572 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.610888 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.610892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.610895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.610956 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.612559 (+ 1603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.612563 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.612659 (+ 96us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.612769 (+ 110us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.612910 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"spinlock_wait_cycles":7168,"thread_start_us":131,"threads_started":1}
I20260128 20:51:26.620671 12332 raft_consensus.cc:493] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:26.620865 12332 raft_consensus.cc:515] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } }
I20260128 20:51:26.622094 12332 leader_election.cc:290] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 28230a487eeb47ca815837c6f045f379 (127.4.183.189:45875)
I20260128 20:51:26.622879 12218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "369959360abb4f4e89afcd777b02bbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28230a487eeb47ca815837c6f045f379" is_pre_election: true
I20260128 20:51:26.622902 11605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "369959360abb4f4e89afcd777b02bbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b783e56be54266bb6628d8f392e2ef" is_pre_election: true
I20260128 20:51:26.623086 12218 raft_consensus.cc:1813] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Handling vote request from an unknown peer 369959360abb4f4e89afcd777b02bbf1
I20260128 20:51:26.623574 12218 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 369959360abb4f4e89afcd777b02bbf1 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260128 20:51:26.624040 12274 leader_election.cc:304] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 369959360abb4f4e89afcd777b02bbf1; no voters: 28230a487eeb47ca815837c6f045f379, 55b783e56be54266bb6628d8f392e2ef
I20260128 20:51:26.624603 12332 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 0 FOLLOWER]: Advancing to term 1
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633484, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.190@KRBTEST.COM for oryx/127.4.183.188@KRBTEST.COM
I20260128 20:51:26.626089 12332 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260128 20:51:26.630360 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.620064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.4.183.188:42553)
0128 20:51:26.620194 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.620197 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.620269 (+ 72us) server_negotiation.cc:408] Connection header received
0128 20:51:26.620595 (+ 326us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.620603 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.620667 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.620785 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.621452 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.621979 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.623173 (+ 1194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.623405 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.625332 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.625352 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.625355 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.625387 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.628119 (+ 2732us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.628964 (+ 845us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.628971 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.628979 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.629044 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.629436 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.629440 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.629442 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.629652 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.629745 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.630107 (+ 362us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.630243 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 20:51:26.630363 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.619965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47172)
0128 20:51:26.620182 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.620199 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.620328 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.620839 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.620842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.621057 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.621325 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.621334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.622104 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.622108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.622727 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.622739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.622888 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.623580 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.623601 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.625183 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.628310 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.628322 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.628325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.628769 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.629230 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.629234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.629236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.629310 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.629769 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.629772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.629884 (+ 112us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.630006 (+ 122us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.630183 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260128 20:51:26.631749 11595 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.4.183.188:42553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "369959360abb4f4e89afcd777b02bbf1"
member_type: NON_VOTER
last_known_addr {
host: "127.4.183.188"
port: 42553
}
attrs {
promote: true
}
}
dest_uuid: "55b783e56be54266bb6628d8f392e2ef"
cas_config_opid_index: 5
I20260128 20:51:26.632624 11595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } }
I20260128 20:51:26.634682 12165 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.635146 12218 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Refusing update from remote peer 55b783e56be54266bb6628d8f392e2ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260128 20:51:26.635227 12300 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Refusing update from remote peer 55b783e56be54266bb6628d8f392e2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260128 20:51:26.635685 12166 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260128 20:51:26.635958 12168 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:26.636487 12165 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.4.183.188:42553
I20260128 20:51:26.636688 12165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.636777 12165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:26.637472 12180 master.cc:561] Master@127.4.183.188:42553 shutting down...
I20260128 20:51:26.638164 12218 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.640586 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b783e56be54266bb6628d8f392e2ef. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.640745 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: LEARNER
I20260128 20:51:26.640848 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.640913 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: LEARNER
I20260128 20:51:26.641042 12167 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: attempting to promote NON_VOTER 28230a487eeb47ca815837c6f045f379 to VOTER
I20260128 20:51:26.641017 12300 raft_consensus.cc:872] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } }
I20260128 20:51:26.641649 12167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } }
I20260128 20:51:26.643509 12217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 LEARNER]: Refusing update from remote peer 55b783e56be54266bb6628d8f392e2ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260128 20:51:26.643931 12168 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260128 20:51:26.644449 11582 consensus_peers.cc:597] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef -> Peer 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188:42553): Couldn't send request to peer 369959360abb4f4e89afcd777b02bbf1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20260128 20:51:26.642516 12300 raft_consensus.cc:1579] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } }
I20260128 20:51:26.646821 12166 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 28230a487eeb47ca815837c6f045f379 (127.4.183.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.647483 12217 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 28230a487eeb47ca815837c6f045f379 (127.4.183.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.649071 12300 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 28230a487eeb47ca815837c6f045f379 (127.4.183.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.650192 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.650309 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:26.650993 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.651120 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:26.651352 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.651450 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:26.651600 12359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.651780 12359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260128 20:51:26.651763 12332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b783e56be54266bb6628d8f392e2ef. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.651969 12332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260128 20:51:26.652069 12358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: NON_VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: true } } }
I20260128 20:51:26.652153 12358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260128 20:51:26.654459 11595 master.cc:679] Remote error: unable to get registration information for peer 369959360abb4f4e89afcd777b02bbf1: Service unavailable: service kudu.master.MasterService not registered on Master
I20260128 20:51:26.654439 12356 mvcc.cc:204] Tried to move back new op lower bound from 7248418754253471744 to 7248418754100580352. Current Snapshot: MvccSnapshot[applied={T|T < 7248418754143887360}]
I20260128 20:51:26.655401 12180 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260128 20:51:26.655723 12180 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260128 20:51:26.655825 12180 tablet_replica.cc:333] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: stopping tablet replica
I20260128 20:51:26.674942 12180 master.cc:583] Master@127.4.183.188:42553 shutdown complete.
I20260128 20:51:26.675027 12180 master_runner.cc:305] Clearing existing system tablet
I20260128 20:51:26.675285 12180 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:26.675352 12180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:26.676894 12180 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260128 20:51:26.678108 12364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:26.678406 12180 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:26.678599 12180 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:26.678651 12180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
uuid: "369959360abb4f4e89afcd777b02bbf1"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "f3c25968a48dbded8b21f61a5c709e27"
server_key_iv: "5cbee00bb638bdec460568ae82a81fcc"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.680397 12180 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:26.682631 12180 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260128 20:51:26.682729 12180 log.cc:1199] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260128 20:51:26.683066 12180 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Deleting consensus metadata
I20260128 20:51:26.683250 12180 master_runner.cc:315] Copying system tablet from 127.4.183.190:45085
I20260128 20:51:26.684556 12180 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.183.190:45085
I20260128 20:51:26.692330 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.691949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47174)
0128 20:51:26.692119 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.692212 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.188:42553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260128 20:51:26.692648 11595 master.cc:679] Network error: unable to get registration information for peer 369959360abb4f4e89afcd777b02bbf1: Client connection negotiation failed: client connection to 127.4.183.188:42553: connect: Connection refused (error 111)
I20260128 20:51:26.695691 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.685216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56866)
0128 20:51:26.685499 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.685523 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.685637 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.685984 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.685987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.686176 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.686399 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.686407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.687553 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.687557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.688080 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.688091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.688226 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.689000 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.689018 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.689484 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.693968 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.693972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.693974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.694186 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.694499 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.694502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.694504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.694574 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.695099 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.695105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.695226 (+ 121us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.695329 (+ 103us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.695487 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
I20260128 20:51:26.695772 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.685348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.4.183.190:45085)
0128 20:51:26.685528 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.685534 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.685586 (+ 52us) server_negotiation.cc:408] Connection header received
0128 20:51:26.685755 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.685759 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.685829 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.685944 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.686557 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.687373 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.688220 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.688466 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.689808 (+ 1342us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.689845 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.689849 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.689903 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.693785 (+ 3882us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.694301 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.694305 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.694308 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.694371 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.694683 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.694687 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.694690 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.694924 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.695041 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.695421 (+ 380us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.695584 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260128 20:51:26.696238 11615 tablet_copy_service.cc:140] P 55b783e56be54266bb6628d8f392e2ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 369959360abb4f4e89afcd777b02bbf1 ({username='oryx', principal='oryx/127.4.183.188@KRBTEST.COM'} at 127.0.0.1:56866)
I20260128 20:51:26.696354 11615 tablet_copy_service.cc:161] P 55b783e56be54266bb6628d8f392e2ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 369959360abb4f4e89afcd777b02bbf1 at {username='oryx', principal='oryx/127.4.183.188@KRBTEST.COM'} at 127.0.0.1:56866: session id = 369959360abb4f4e89afcd777b02bbf1-00000000000000000000000000000000
I20260128 20:51:26.697592 11615 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260128 20:51:26.699151 12180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260128 20:51:26.701088 12180 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: tablet copy: Starting download of 0 data blocks...
I20260128 20:51:26.701401 12180 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: tablet copy: Starting download of 1 WAL segments...
I20260128 20:51:26.703547 12180 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260128 20:51:26.705524 11615 tablet_copy_service.cc:342] P 55b783e56be54266bb6628d8f392e2ef: Request end of tablet copy session 369959360abb4f4e89afcd777b02bbf1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.183.188@KRBTEST.COM'} at 127.0.0.1:56866
I20260128 20:51:26.705659 11615 tablet_copy_service.cc:434] P 55b783e56be54266bb6628d8f392e2ef: ending tablet copy session 369959360abb4f4e89afcd777b02bbf1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 369959360abb4f4e89afcd777b02bbf1
W20260128 20:51:26.712608 12180 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260128 20:51:26.715801 12180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:26.718228 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:26.718287 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:26.718498 12180 server_base.cc:1060] running on GCE node
W20260128 20:51:26.718614 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:26.718834 12180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:26.719187 12180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:26.720412 12180 hybrid_clock.cc:648] HybridClock initialized: now 1769633486720387 us; error 48 us; skew 500 ppm
Jan 28 20:51:26 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633486, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:26.722468 12180 init.cc:377] Logged in from keytab as oryx/127.4.183.188@KRBTEST.COM (short username oryx)
I20260128 20:51:26.723133 12180 webserver.cc:492] Webserver started at http://127.4.183.188:46229/ using document root <none> and password file <none>
I20260128 20:51:26.723317 12180 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:26.723351 12180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:26.724526 12180 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:26.725580 12381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:26.725839 12180 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:26.725920 12180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
uuid: "369959360abb4f4e89afcd777b02bbf1"
format_stamp: "Formatted at 2026-01-28 20:51:26 on dist-test-slave-hss7"
server_key: "f3c25968a48dbded8b21f61a5c709e27"
server_key_iv: "5cbee00bb638bdec460568ae82a81fcc"
server_key_version: "encryptionkey@0"
I20260128 20:51:26.725993 12180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:26.738241 12180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:26.740995 12180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:26.747593 12180 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.188:42553
I20260128 20:51:26.747859 12433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.188:42553 every 8 connection(s)
I20260128 20:51:26.748325 12180 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/master-2/data/info.pb
I20260128 20:51:26.751012 12434 sys_catalog.cc:263] Verifying existing consensus state
I20260128 20:51:26.752259 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Bootstrap starting.
I20260128 20:51:26.764883 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260128 20:51:26.765532 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Bootstrap complete.
I20260128 20:51:26.766355 12434 raft_consensus.cc:359] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } }
I20260128 20:51:26.766511 12434 raft_consensus.cc:740] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 369959360abb4f4e89afcd777b02bbf1, State: Initialized, Role: LEARNER
I20260128 20:51:26.766630 12434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } }
I20260128 20:51:26.767226 12434 sys_catalog.cc:565] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:26.767164 12438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:26.767732 12438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: LEARNER
I20260128 20:51:26.770565 12449 catalog_manager.cc:1269] Loaded cluster ID: 4fbd71821df24198bcc7e6c817ea9858
I20260128 20:51:26.770709 12449 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: loading cluster ID for follower catalog manager: success
I20260128 20:51:26.771296 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.761224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47182)
0128 20:51:26.761385 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:26.761403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:26.761499 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:26.762128 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:26.762131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:26.762356 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:26.762544 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.762552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.763503 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.763506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:26.763886 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:26.763893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.764038 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.764785 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:26.764809 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:26.765509 (+ 700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:26.768826 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.768831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.768833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.769126 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.770090 (+ 964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:26.770095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:26.770097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:26.770175 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:26.770706 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:26.770709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:26.770849 (+ 140us) client_negotiation.cc:769] Sending connection context
0128 20:51:26.770961 (+ 112us) client_negotiation.cc:240] Negotiation successful
0128 20:51:26.771128 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260128 20:51:26.771893 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:26.761448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.4.183.188:42553)
0128 20:51:26.761756 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:26.761760 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:26.761771 (+ 11us) server_negotiation.cc:408] Connection header received
0128 20:51:26.761812 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:26.761815 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:26.761954 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:26.762109 (+ 155us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:26.762681 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.763372 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:26.764014 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:26.764223 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:26.765822 (+ 1599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:26.765852 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:26.765855 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:26.765908 (+ 53us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:26.768643 (+ 2735us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.769256 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.769260 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.769262 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.769326 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:26.770308 (+ 982us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:26.770312 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:26.770314 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:26.770544 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:26.770641 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:26.771569 (+ 928us) server_negotiation.cc:300] Negotiation successful
0128 20:51:26.771704 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":148,"threads_started":1}
I20260128 20:51:26.773427 12449 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: acquiring CA information for follower catalog manager: success
I20260128 20:51:26.775079 12449 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260128 20:51:26.834653 12060 heartbeater.cc:499] Master 127.4.183.190:45085 was elected leader, sending a full tablet report...
I20260128 20:51:27.060359 12407 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260128 20:51:27.060864 12438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b783e56be54266bb6628d8f392e2ef. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:27.061045 12438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: LEARNER
I20260128 20:51:27.495101 12167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: NON_VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: true } } }
I20260128 20:51:27.495339 12167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:27.496749 12167 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: attempting to promote NON_VOTER 369959360abb4f4e89afcd777b02bbf1 to VOTER
I20260128 20:51:27.497660 12167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:27.499442 12217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Refusing update from remote peer 55b783e56be54266bb6628d8f392e2ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260128 20:51:27.499444 12407 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 LEARNER]: Refusing update from remote peer 55b783e56be54266bb6628d8f392e2ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260128 20:51:27.500205 12450 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260128 20:51:27.500734 12167 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260128 20:51:27.503407 12452 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.504114 12217 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.504206 12407 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.505080 12452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.505223 12452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:27.505414 12452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.505484 12452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:27.505553 12452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.505594 12452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:27.505760 12438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.505883 12438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:27.506211 12454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b783e56be54266bb6628d8f392e2ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:27.506340 12454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:27.699414 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.690248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.4.183.189:45875)
0128 20:51:27.690594 (+ 346us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.690599 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.690695 (+ 96us) server_negotiation.cc:408] Connection header received
0128 20:51:27.690838 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.690841 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.690892 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.690981 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:27.691942 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.692484 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.693375 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.693839 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.695245 (+ 1406us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:27.695272 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:27.695274 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:27.695315 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:27.697190 (+ 1875us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.697744 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.697748 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.697749 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.697808 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.698156 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.698161 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.698163 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.698452 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:27.698672 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.699031 (+ 359us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.699239 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260128 20:51:27.703210 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.700307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56872)
0128 20:51:27.700570 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.700583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.700666 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.700987 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.700989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.701006 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:27.701296 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.701303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.702197 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.702201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.702852 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.702861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.702969 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.702992 (+ 23us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.703031 (+ 39us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.703086 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260128 20:51:27.703711 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.700503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.4.183.190:45085)
0128 20:51:27.700759 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.700762 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.700775 (+ 13us) server_negotiation.cc:408] Connection header received
0128 20:51:27.700826 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.700828 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.700877 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.700957 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:27.701448 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.702059 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.703000 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.703396 (+ 396us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.703451 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.703508 (+ 57us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.703607 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
I20260128 20:51:27.708130 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.705210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39328)
0128 20:51:27.705347 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.705368 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.705444 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.705743 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.705746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.705767 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:27.705954 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.705960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.707097 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.707100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.707746 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.707755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.707894 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.707913 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.707953 (+ 40us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.708012 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 20:51:27.708691 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.705272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.4.183.189:45875)
0128 20:51:27.705406 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.705409 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.705423 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:27.705532 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.705535 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.705613 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.705689 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:27.706189 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.706899 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.707948 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.708384 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.708426 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.708532 (+ 106us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.708597 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 20:51:27.713336 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.710188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47186)
0128 20:51:27.710346 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.710359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.710431 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.711164 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.711167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.711186 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:27.711354 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.711360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.712262 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.712265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.712907 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.712915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.713020 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.713039 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.713076 (+ 37us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.713212 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260128 20:51:27.713999 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.710395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.4.183.188:42553)
0128 20:51:27.710720 (+ 325us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.710723 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.710737 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:27.710939 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.710942 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.711006 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.711114 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:27.711520 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.712132 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.713051 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.713631 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.713785 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.713835 (+ 50us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.713894 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":142,"threads_started":1}
I20260128 20:51:27.725664 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.716561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.4.183.188:42553)
0128 20:51:27.716711 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.716715 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.716727 (+ 12us) server_negotiation.cc:408] Connection header received
0128 20:51:27.716784 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.716787 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.716828 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.716905 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:27.718291 (+ 1386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.719001 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.720282 (+ 1281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.720500 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.721950 (+ 1450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:27.721974 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:27.721976 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:27.722010 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:27.723803 (+ 1793us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.724336 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.724339 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.724341 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.724392 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.724709 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.724712 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.724713 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.724902 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:27.725021 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.725380 (+ 359us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.725526 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 20:51:27.729633 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.726727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:56884)
0128 20:51:27.726993 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.727006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.727094 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.727424 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.727427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.727460 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:27.727694 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.727700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.728534 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.728537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.729256 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.729266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.729380 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.729402 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.729442 (+ 40us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.729498 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":107,"threads_started":1}
I20260128 20:51:27.730072 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.726878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.4.183.190:45085)
0128 20:51:27.727031 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.727035 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.727049 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:27.727233 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.727236 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.727285 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.727363 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:27.727868 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.728389 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.729394 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.729807 (+ 413us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.729851 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.729906 (+ 55us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.729980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 20:51:27.734129 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.731391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.0.0.1:39330)
0128 20:51:27.731525 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.731548 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.731625 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.731947 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.731949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.731970 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:27.732134 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.732139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.732877 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.732880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.733607 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.733620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.733741 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.733763 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.733806 (+ 43us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.733865 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260128 20:51:27.734735 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.731597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.4.183.189:45875)
0128 20:51:27.731726 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.731730 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.731742 (+ 12us) server_negotiation.cc:408] Connection header received
0128 20:51:27.731788 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.731790 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.731842 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.731912 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:27.732231 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.732735 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.733856 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.734317 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.734440 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.734519 (+ 79us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.734609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 20:51:27.740494 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.736760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.0.0.1:47208)
0128 20:51:27.736946 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.736966 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.737112 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.737650 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.737656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.737696 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:27.738043 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.738053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.739464 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.739467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.740069 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.740079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.740213 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.740246 (+ 33us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.740291 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.740361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260128 20:51:27.741365 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.736969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.4.183.188:42553)
0128 20:51:27.737302 (+ 333us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.737312 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.737332 (+ 20us) server_negotiation.cc:408] Connection header received
0128 20:51:27.737399 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.737405 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.737468 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.737583 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:27.738268 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.739251 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.740264 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.740989 (+ 725us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.741039 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.741093 (+ 54us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.741243 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260128 20:51:27.743348 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11659
I20260128 20:51:27.751477 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:46141
--local_ip_for_outbound_sockets=127.4.183.129
--tserver_master_addrs=127.4.183.190:45085
--webserver_port=33121
--webserver_interface=127.4.183.129
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:27.894826 12461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:27.895119 12461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:27.895200 12461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:27.899138 12461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:27.899253 12461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:27.899341 12461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:27.904284 12461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:46141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=33121
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:27.905503 12461 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:27.906368 12461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:27.915493 12467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:27.915478 12469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:27.915988 12461 server_base.cc:1060] running on GCE node
W20260128 20:51:27.915489 12466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:27.916757 12461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:27.918431 12461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:27.921922 12461 hybrid_clock.cc:648] HybridClock initialized: now 1769633487921841 us; error 123 us; skew 500 ppm
Jan 28 20:51:27 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633487, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:27.925263 12461 init.cc:377] Logged in from keytab as oryx/127.4.183.129@KRBTEST.COM (short username oryx)
I20260128 20:51:27.926539 12461 webserver.cc:492] Webserver started at http://127.4.183.129:33121/ using document root <none> and password file <none>
I20260128 20:51:27.927157 12461 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:27.927234 12461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:27.931674 12461 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:27.934664 12476 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:27.936226 12461 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:27.936417 12461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "353acb53f731489badbeb29923c2f9dc"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "51ad8665831f934fffa0fc67556d8376"
server_key_iv: "106607897d6924625b4c7ad898bf68be"
server_key_version: "encryptionkey@0"
I20260128 20:51:27.936983 12461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:27.953747 12461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:27.957468 12461 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:27.957720 12461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:27.958397 12461 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:27.959483 12461 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:27.959537 12461 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:27.959590 12461 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:27.959606 12461 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:27.970989 12461 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:46141
I20260128 20:51:27.971014 12589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:46141 every 8 connection(s)
I20260128 20:51:27.972270 12461 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:27.972666 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 12461
I20260128 20:51:27.972893 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11795
I20260128 20:51:27.983660 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:45699
--local_ip_for_outbound_sockets=127.4.183.130
--tserver_master_addrs=127.4.183.190:45085
--webserver_port=33685
--webserver_interface=127.4.183.130
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 28 20:51:27 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633487, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.188@KRBTEST.COM
Jan 28 20:51:27 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633487, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
Jan 28 20:51:27 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633487, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
I20260128 20:51:27.992977 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.976293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:47903 (local address 127.4.183.188:42553)
0128 20:51:27.976438 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.976445 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.976693 (+ 248us) server_negotiation.cc:408] Connection header received
0128 20:51:27.977306 (+ 613us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.977313 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.977410 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.977480 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:27.980817 (+ 3337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.981867 (+ 1050us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.983019 (+ 1152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.983233 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.987612 (+ 4379us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:27.987648 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:27.987655 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:27.987701 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:27.989985 (+ 2284us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.990917 (+ 932us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.990923 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.990925 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.990993 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.991442 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.991446 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.991447 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.991706 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:27.991783 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.992612 (+ 829us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.992784 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260128 20:51:27.993441 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.976771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:57833 (local address 127.4.183.189:45875)
0128 20:51:27.976938 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.976945 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.977349 (+ 404us) server_negotiation.cc:408] Connection header received
0128 20:51:27.977559 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.977565 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.977626 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.977677 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:27.980847 (+ 3170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.981793 (+ 946us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.982962 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.983185 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.989079 (+ 5894us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:27.989110 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:27.989113 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:27.989199 (+ 86us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:27.991555 (+ 2356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.992109 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.992113 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.992115 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.992167 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.992434 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.992437 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.992439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.992641 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:27.992721 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.993079 (+ 358us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.993260 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 20:51:27.994318 12596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.976663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.4.183.129:57833)
0128 20:51:27.977272 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.977288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.977441 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.977994 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.978008 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.979121 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:27.980455 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.980496 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.982039 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.982044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.982793 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.982804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.983650 (+ 846us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.984724 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:27.984742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:27.988789 (+ 4047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:27.991736 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:27.991742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:27.991746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:27.991985 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:27.992280 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:27.992282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:27.992284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:27.992329 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:27.992810 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:27.992813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:27.992893 (+ 80us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.993645 (+ 752us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.993858 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"spinlock_wait_cycles":48896,"thread_start_us":147,"threads_started":1}
I20260128 20:51:27.994509 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.975059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:49389)
0128 20:51:27.975747 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.975811 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.977002 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.977994 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.978008 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.979122 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:27.980455 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.980496 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.982039 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.982044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.982791 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.982804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.983650 (+ 846us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.984465 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:27.984535 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:27.987567 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:27.991178 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:27.991186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:27.991191 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:27.991462 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:27.991846 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:27.991850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:27.991852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:27.991940 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:27.992498 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:27.992503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:27.992626 (+ 123us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.992785 (+ 159us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.993194 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"spinlock_wait_cycles":75776,"thread_start_us":142,"threads_started":1}
I20260128 20:51:27.994346 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.976178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.4.183.129:47903)
0128 20:51:27.976592 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:27.976617 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:27.977006 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:27.977995 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:27.978010 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:27.979308 (+ 1298us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:27.980456 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.980500 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.982118 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.982125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:27.982839 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:27.982865 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.983577 (+ 712us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.984600 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:27.984624 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:27.987326 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:27.990198 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:27.990208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:27.990227 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:27.990742 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:27.991131 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:27.991143 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:27.991148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:27.991305 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:27.991871 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:27.991880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:27.992274 (+ 394us) client_negotiation.cc:769] Sending connection context
0128 20:51:27.993682 (+ 1408us) client_negotiation.cc:240] Negotiation successful
0128 20:51:27.993935 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":130,"spinlock_wait_cycles":7424,"thread_start_us":102,"threads_started":1}
I20260128 20:51:27.995805 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:27.974653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:49389 (local address 127.4.183.190:45085)
0128 20:51:27.974843 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:27.974856 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:27.975981 (+ 1125us) server_negotiation.cc:408] Connection header received
0128 20:51:27.977333 (+ 1352us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:27.977339 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:27.977426 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:27.977488 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:27.980837 (+ 3349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.981793 (+ 956us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:27.982962 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:27.983185 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:27.987784 (+ 4599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:27.987816 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:27.987823 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:27.987870 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:27.990968 (+ 3098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.991634 (+ 666us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.991642 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.991645 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.991719 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:27.992070 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:27.992075 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:27.992080 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:27.992331 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:27.992443 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:27.994956 (+ 2513us) server_negotiation.cc:300] Negotiation successful
0128 20:51:27.995599 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260128 20:51:27.996057 12592 heartbeater.cc:344] Connected to a master server at 127.4.183.189:45875
I20260128 20:51:27.996340 12592 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:27.996389 12591 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45085
I20260128 20:51:27.996490 12591 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:27.997030 12591 heartbeater.cc:507] Master 127.4.183.190:45085 requested a full tablet report, sending...
I20260128 20:51:27.997084 12590 heartbeater.cc:344] Connected to a master server at 127.4.183.188:42553
I20260128 20:51:27.997030 12592 heartbeater.cc:507] Master 127.4.183.189:45875 requested a full tablet report, sending...
I20260128 20:51:27.997215 12590 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:27.997423 12590 heartbeater.cc:507] Master 127.4.183.188:42553 requested a full tablet report, sending...
I20260128 20:51:27.998574 11595 ts_manager.cc:194] Re-registered known tserver with Master: 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141)
I20260128 20:51:27.999387 12398 ts_manager.cc:194] Registered new tserver with Master: 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141)
I20260128 20:51:27.999382 12208 ts_manager.cc:194] Registered new tserver with Master: 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141)
I20260128 20:51:27.999418 11595 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:49389
W20260128 20:51:28.128892 12597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:28.129272 12597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:28.129376 12597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:28.133663 12597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:28.133793 12597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:28.133919 12597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:28.139149 12597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:45699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=33685
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:28.140431 12597 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:28.141508 12597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:28.149709 12603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:28.149711 12602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:28.149726 12605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:28.150205 12597 server_base.cc:1060] running on GCE node
I20260128 20:51:28.150708 12597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:28.151413 12597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:28.152887 12597 hybrid_clock.cc:648] HybridClock initialized: now 1769633488152844 us; error 59 us; skew 500 ppm
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:28.157037 12597 init.cc:377] Logged in from keytab as oryx/127.4.183.130@KRBTEST.COM (short username oryx)
I20260128 20:51:28.158530 12597 webserver.cc:492] Webserver started at http://127.4.183.130:33685/ using document root <none> and password file <none>
I20260128 20:51:28.159521 12597 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:28.159658 12597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:28.165306 12597 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260128 20:51:28.168313 12612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:28.169977 12597 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260128 20:51:28.170154 12597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "fcceb005c7c548679dd16848295150e7"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "e8e906c01dfa74f8b852427398b4819b"
server_key_iv: "7ca2845100b6bae07a0d73981c5cf27d"
server_key_version: "encryptionkey@0"
I20260128 20:51:28.170749 12597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:28.184568 12597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:28.188650 12597 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:28.188839 12597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:28.189646 12597 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:28.190727 12597 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:28.190783 12597 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:28.190824 12597 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:28.190840 12597 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:28.205132 12597 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:45699
I20260128 20:51:28.205184 12725 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:45699 every 8 connection(s)
I20260128 20:51:28.206321 12597 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:28.213290 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 12597
I20260128 20:51:28.213490 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11931
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.188@KRBTEST.COM
I20260128 20:51:28.224112 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.208278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:41391 (local address 127.4.183.190:45085)
0128 20:51:28.208441 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.208445 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.209300 (+ 855us) server_negotiation.cc:408] Connection header received
0128 20:51:28.211023 (+ 1723us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.211029 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.211100 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.211193 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.212465 (+ 1272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.213012 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.217860 (+ 4848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.218130 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.218943 (+ 813us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.218971 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.218978 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.219012 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.220835 (+ 1823us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.222142 (+ 1307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.222147 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.222148 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.222208 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.222612 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.222617 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.222620 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.222871 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.223003 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.223724 (+ 721us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.223908 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260128 20:51:28.224925 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.208604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:41391)
0128 20:51:28.209111 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.209188 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.210328 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.211232 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.211235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.211556 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:28.212321 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.212330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.213179 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.213183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.214645 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.214667 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.214835 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.216199 (+ 1364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:28.216225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:28.218771 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:28.221431 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.221441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.221462 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.222007 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.222328 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.222334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.222339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.222502 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.223009 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:28.223021 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:28.223397 (+ 376us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.223667 (+ 270us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.224008 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"spinlock_wait_cycles":123776,"thread_start_us":168,"threads_started":1}
I20260128 20:51:28.224963 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.208550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:52359 (local address 127.4.183.189:45875)
0128 20:51:28.208705 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.208710 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.209295 (+ 585us) server_negotiation.cc:408] Connection header received
0128 20:51:28.210532 (+ 1237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.210535 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.210607 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.210688 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.212496 (+ 1808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.213074 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.214254 (+ 1180us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.214491 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.218113 (+ 3622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.218141 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.218144 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.218183 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.221245 (+ 3062us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.222368 (+ 1123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.222374 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.222377 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.222442 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.223231 (+ 789us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.223235 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.223237 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.223455 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.223541 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.224626 (+ 1085us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.224789 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260128 20:51:28.225229 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:40173
--local_ip_for_outbound_sockets=127.4.183.131
--tserver_master_addrs=127.4.183.190:45085
--webserver_port=41199
--webserver_interface=127.4.183.131
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553 with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 20:51:28.225548 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.208595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.4.183.130:52359)
0128 20:51:28.209111 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.209190 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.210310 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.210883 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.210892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.211555 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:28.212320 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.212330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.213268 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.213273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.214092 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.214108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.214833 (+ 725us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.215652 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:28.215711 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:28.217860 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:28.221437 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.221441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.221460 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.221998 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.223035 (+ 1037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.223042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.223046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.223109 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.224289 (+ 1180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:28.224295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:28.224398 (+ 103us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.225107 (+ 709us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.225295 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"spinlock_wait_cycles":54016,"thread_start_us":150,"threads_started":1}
I20260128 20:51:28.227160 12727 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45085
I20260128 20:51:28.227512 12727 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:28.228148 12728 heartbeater.cc:344] Connected to a master server at 127.4.183.189:45875
I20260128 20:51:28.228236 12728 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:28.228365 12727 heartbeater.cc:507] Master 127.4.183.190:45085 requested a full tablet report, sending...
I20260128 20:51:28.228395 12728 heartbeater.cc:507] Master 127.4.183.189:45875 requested a full tablet report, sending...
I20260128 20:51:28.229544 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.209936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:34561 (local address 127.4.183.188:42553)
0128 20:51:28.210073 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.210077 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.210220 (+ 143us) server_negotiation.cc:408] Connection header received
0128 20:51:28.210697 (+ 477us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.210700 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.210748 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.210844 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.213221 (+ 2377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.214108 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.219015 (+ 4907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.219258 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.222171 (+ 2913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.222198 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.222201 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.222238 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.224428 (+ 2190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.228182 (+ 3754us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.228187 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.228191 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.228272 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.228557 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.228562 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.228564 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.228739 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.228874 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.229222 (+ 348us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.229379 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 20:51:28.229794 11595 ts_manager.cc:194] Re-registered known tserver with Master: fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:28.229810 12208 ts_manager.cc:194] Registered new tserver with Master: fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:28.230074 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.209791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.4.183.130:34561)
0128 20:51:28.210125 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.210140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.210335 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.210882 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.210893 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.211821 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:28.212279 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.212293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.218273 (+ 5980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.218277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.218808 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.218816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.219421 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.220016 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:28.220035 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:28.221998 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:28.227658 (+ 5660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.227665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.227670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.228001 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.228403 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.228405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.228407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.228452 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.228861 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:28.228865 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:28.228986 (+ 121us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.229746 (+ 760us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.229908 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260128 20:51:28.230563 11595 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:41391
I20260128 20:51:28.230693 12726 heartbeater.cc:344] Connected to a master server at 127.4.183.188:42553
I20260128 20:51:28.230804 12726 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:28.231031 12726 heartbeater.cc:507] Master 127.4.183.188:42553 requested a full tablet report, sending...
I20260128 20:51:28.231992 12398 ts_manager.cc:194] Registered new tserver with Master: fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
W20260128 20:51:28.361068 12733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:28.361443 12733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:28.361510 12733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:28.365092 12733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:28.365302 12733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:28.365490 12733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:28.370577 12733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:40173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=41199
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:45085,127.4.183.189:45875,127.4.183.188:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:28.371917 12733 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:28.372844 12733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:28.382412 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:28.382412 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:28.383292 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:28.383582 12733 server_base.cc:1060] running on GCE node
I20260128 20:51:28.384027 12733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:28.384645 12733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:28.385843 12733 hybrid_clock.cc:648] HybridClock initialized: now 1769633488385824 us; error 46 us; skew 500 ppm
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:28.389662 12733 init.cc:377] Logged in from keytab as oryx/127.4.183.131@KRBTEST.COM (short username oryx)
I20260128 20:51:28.391031 12733 webserver.cc:492] Webserver started at http://127.4.183.131:41199/ using document root <none> and password file <none>
I20260128 20:51:28.391700 12733 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:28.391773 12733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:28.397814 12733 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:51:28.401057 12748 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:28.403055 12733 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:51:28.403276 12733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "27e7ed4ed52848ce89ffdc349ae45ca5"
format_stamp: "Formatted at 2026-01-28 20:51:25 on dist-test-slave-hss7"
server_key: "cbeef5355f6491dc07091c051b2a72db"
server_key_iv: "667b3e8191f4433710759732260bb8a8"
server_key_version: "encryptionkey@0"
I20260128 20:51:28.404042 12733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:28.429418 12733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:28.433636 12733 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:28.433888 12733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:28.434577 12733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:28.435823 12733 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:28.435906 12733 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:28.435985 12733 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:28.436028 12733 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:28.447952 12733 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:40173
I20260128 20:51:28.447992 12861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:40173 every 8 connection(s)
I20260128 20:51:28.449214 12733 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:28.458016 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 12733
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.188@KRBTEST.COM
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:28.468392 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.453502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:43411 (local address 127.4.183.188:42553)
0128 20:51:28.453861 (+ 359us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.453867 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.454027 (+ 160us) server_negotiation.cc:408] Connection header received
0128 20:51:28.454275 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.454279 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.454346 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.454423 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.456852 (+ 2429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.457534 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.458705 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.458958 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.462668 (+ 3710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.462688 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.462691 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.462726 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.465388 (+ 2662us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.466265 (+ 877us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.466269 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.466271 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.466334 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.466718 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.466724 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.466726 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.466954 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.467070 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.468051 (+ 981us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.468176 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Jan 28 20:51:28 dist-test-slave-hss7 krb5kdc[11547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633488, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.189@KRBTEST.COM
I20260128 20:51:28.470337 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.453254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.188:42553 (local address 127.4.183.131:43411)
0128 20:51:28.453859 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.453890 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.454130 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.454653 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.454657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.455583 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:28.456574 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.456598 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.457868 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.457875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.458502 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.458518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.459308 (+ 790us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.460084 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:28.460141 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:28.462469 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:28.465542 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.465550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.465574 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.466124 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.466445 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.466450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.466452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.466614 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.467070 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:28.467082 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:28.467489 (+ 407us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.467791 (+ 302us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.469284 (+ 1493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"spinlock_wait_cycles":8960,"thread_start_us":123,"threads_started":1}
I20260128 20:51:28.471905 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.451960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:46695)
0128 20:51:28.452521 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.452580 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.453876 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.454872 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.454876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.455889 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:28.456594 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.456609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.458413 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.458417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.459091 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.459104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.459392 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.461087 (+ 1695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:28.461110 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:28.463006 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:28.465702 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.465708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.465712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.466124 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.466736 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.466739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.466741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.466787 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.471129 (+ 4342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:28.471133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:28.471262 (+ 129us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.471392 (+ 130us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.471596 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":36,"spinlock_wait_cycles":56832,"thread_start_us":176,"threads_started":1}
I20260128 20:51:28.472325 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.451673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:46695 (local address 127.4.183.190:45085)
0128 20:51:28.451837 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.451842 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.452709 (+ 867us) server_negotiation.cc:408] Connection header received
0128 20:51:28.454634 (+ 1925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.454639 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.454713 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.454810 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.457352 (+ 2542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.458039 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.459557 (+ 1518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.460319 (+ 762us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.463141 (+ 2822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.463167 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.463176 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.463209 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.465269 (+ 2060us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.466478 (+ 1209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.466482 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.466484 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.466546 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.467565 (+ 1019us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.467569 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.467571 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.467794 (+ 223us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.467901 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.472024 (+ 4123us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.472173 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260128 20:51:28.472354 12862 heartbeater.cc:344] Connected to a master server at 127.4.183.188:42553
I20260128 20:51:28.472702 12862 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:28.472918 12863 heartbeater.cc:344] Connected to a master server at 127.4.183.190:45085
I20260128 20:51:28.473006 12863 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:28.473745 12863 heartbeater.cc:507] Master 127.4.183.190:45085 requested a full tablet report, sending...
I20260128 20:51:28.474472 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.451720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:60351 (local address 127.4.183.189:45875)
0128 20:51:28.451858 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.451863 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.452705 (+ 842us) server_negotiation.cc:408] Connection header received
0128 20:51:28.454188 (+ 1483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.454196 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.454279 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.454379 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.458369 (+ 3990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.459223 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.462721 (+ 3498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.462964 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.469782 (+ 6818us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.469806 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.469809 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.469854 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.471784 (+ 1930us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.473265 (+ 1481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.473269 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.473272 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.473334 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.473611 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.473623 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.473625 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.473791 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.473915 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.474209 (+ 294us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.474333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":1408}
I20260128 20:51:28.475299 11595 ts_manager.cc:194] Re-registered known tserver with Master: 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173)
I20260128 20:51:28.475468 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.460457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.4.183.189:45875)
0128 20:51:28.460747 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.460752 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.461162 (+ 410us) server_negotiation.cc:408] Connection header received
0128 20:51:28.461391 (+ 229us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.461398 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.461473 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.461548 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.462815 (+ 1267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.463737 (+ 922us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.464788 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.464994 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.467034 (+ 2040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.467065 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.467071 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.467114 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.469277 (+ 2163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.469812 (+ 535us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.469816 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.469818 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.469898 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.470497 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.470500 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.470502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.470715 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.470826 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.475121 (+ 4295us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.475264 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"spinlock_wait_cycles":10880,"thread_start_us":90,"threads_started":1}
I20260128 20:51:28.475927 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.461715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.4.183.188:42553)
0128 20:51:28.462330 (+ 615us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.462335 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.463737 (+ 1402us) server_negotiation.cc:408] Connection header received
0128 20:51:28.463789 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.463794 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.463870 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.463955 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.465088 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.465906 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.467140 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.467378 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.471094 (+ 3716us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.471123 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.471129 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.471168 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.473193 (+ 2025us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.473731 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.473734 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.473736 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.473788 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.474071 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.474075 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.474077 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.474850 (+ 773us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.474987 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.475631 (+ 644us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.475766 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":97,"threads_started":1}
I20260128 20:51:28.476063 11595 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:46695
I20260128 20:51:28.474725 12862 heartbeater.cc:507] Master 127.4.183.188:42553 requested a full tablet report, sending...
I20260128 20:51:28.474483 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.451960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.189:45875 (local address 127.4.183.131:60351)
0128 20:51:28.452521 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.452580 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.453876 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.454598 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.454610 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.455577 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:28.456594 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.456609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.460475 (+ 3866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.460481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.462578 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.462588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.463481 (+ 893us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.465790 (+ 2309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:28.465806 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:28.469625 (+ 3819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:28.472709 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.472713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.472715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.473094 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.473451 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:28.473454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:28.473456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:28.473516 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:28.473928 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:28.473933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:28.474044 (+ 111us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.474148 (+ 104us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.474304 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"spinlock_wait_cycles":58624,"thread_start_us":143,"threads_started":1}
I20260128 20:51:28.477854 12864 heartbeater.cc:344] Connected to a master server at 127.4.183.189:45875
I20260128 20:51:28.477988 12864 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:28.478826 12864 heartbeater.cc:507] Master 127.4.183.189:45875 requested a full tablet report, sending...
I20260128 20:51:28.479890 12208 ts_manager.cc:194] Registered new tserver with Master: 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173)
I20260128 20:51:28.480597 12398 ts_manager.cc:194] Registered new tserver with Master: 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173)
I20260128 20:51:28.482149 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.460269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.4.183.190:45085)
0128 20:51:28.468706 (+ 8437us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.468711 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.468724 (+ 13us) server_negotiation.cc:408] Connection header received
0128 20:51:28.468769 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.468775 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.468847 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.469976 (+ 1129us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.470109 (+ 133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.470918 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.477038 (+ 6120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.477286 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.477601 (+ 315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.477619 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.477622 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.477648 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.480241 (+ 2593us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.480895 (+ 654us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.480899 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.480900 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.480958 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.481309 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.481312 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.481313 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.481488 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.481599 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.481888 (+ 289us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.482011 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8308,"thread_start_us":107,"threads_started":1}
I20260128 20:51:28.485464 11595 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:28.488082 11595 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:28.507349 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.502689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:40173 (local address 127.0.0.1:33552)
0128 20:51:28.503004 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.503021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.503120 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.503867 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.503870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.503890 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:28.504192 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.504204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.505920 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.505925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.506889 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.506900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.507024 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.507048 (+ 24us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.507102 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.507173 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":148,"threads_started":1}
I20260128 20:51:28.508097 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.502904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.4.183.131:40173)
0128 20:51:28.503319 (+ 415us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.503329 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.503356 (+ 27us) server_negotiation.cc:408] Connection header received
0128 20:51:28.503462 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.503473 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.503703 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.503892 (+ 189us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:28.504327 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.505266 (+ 939us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.507012 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.507600 (+ 588us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.507742 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.507838 (+ 96us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.507950 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":92,"threads_started":1}
I20260128 20:51:28.508291 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.503395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:45699 (local address 127.0.0.1:42826)
0128 20:51:28.505462 (+ 2067us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.505475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.505618 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.506049 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.506053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.506071 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:28.506301 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.506308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.507244 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.507248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.507881 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.507893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.508009 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.508028 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.508078 (+ 50us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.508145 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1999,"thread_start_us":95,"threads_started":1}
I20260128 20:51:28.509534 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.503569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.4.183.130:45699)
0128 20:51:28.504180 (+ 611us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.504185 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.505567 (+ 1382us) server_negotiation.cc:408] Connection header received
0128 20:51:28.505765 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.505769 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.505909 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.506051 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:28.506419 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.507121 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.508023 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.509030 (+ 1007us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.509224 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.509293 (+ 69us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.509389 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":119,"threads_started":1}
I20260128 20:51:28.510974 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.503086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:46141 (local address 127.0.0.1:49050)
0128 20:51:28.504146 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.504161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.505206 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.505741 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.505744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.505763 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:28.506044 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.506051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.509741 (+ 3690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.509746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.510501 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.510514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.510654 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.510679 (+ 25us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.510731 (+ 52us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.510810 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":90,"threads_started":1}
I20260128 20:51:28.511758 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.503808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.4.183.129:46141)
0128 20:51:28.504086 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.504093 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.504256 (+ 163us) server_negotiation.cc:408] Connection header received
0128 20:51:28.505410 (+ 1154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.505415 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.505598 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.505763 (+ 165us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:28.506171 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.509550 (+ 3379us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.510687 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.511267 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.511405 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.511484 (+ 79us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.511600 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260128 20:51:28.511725 12796 tablet_service.cc:1505] Processing CreateTablet for tablet 182c4772e6a24daa8358b4d4bf35a53c (DEFAULT_TABLE table=test-table [id=2b3da41e68ed4ef788232e08bacbc20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:28.513610 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182c4772e6a24daa8358b4d4bf35a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:28.513389 12660 tablet_service.cc:1505] Processing CreateTablet for tablet 182c4772e6a24daa8358b4d4bf35a53c (DEFAULT_TABLE table=test-table [id=2b3da41e68ed4ef788232e08bacbc20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:28.514444 12660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182c4772e6a24daa8358b4d4bf35a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:28.515632 12524 tablet_service.cc:1505] Processing CreateTablet for tablet 182c4772e6a24daa8358b4d4bf35a53c (DEFAULT_TABLE table=test-table [id=2b3da41e68ed4ef788232e08bacbc20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:28.516711 12524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182c4772e6a24daa8358b4d4bf35a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:28.521170 12893 tablet_bootstrap.cc:492] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Bootstrap starting.
I20260128 20:51:28.524140 12893 tablet_bootstrap.cc:654] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:28.524960 12893 log.cc:826] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:28.525604 12894 tablet_bootstrap.cc:492] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Bootstrap starting.
I20260128 20:51:28.525636 12895 tablet_bootstrap.cc:492] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Bootstrap starting.
I20260128 20:51:28.527478 12893 tablet_bootstrap.cc:492] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: No bootstrap required, opened a new log
I20260128 20:51:28.527724 12893 ts_tablet_manager.cc:1403] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260128 20:51:28.528167 12895 tablet_bootstrap.cc:654] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:28.528294 12894 tablet_bootstrap.cc:654] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:28.529084 12895 log.cc:826] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:28.529455 12894 log.cc:826] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:28.531559 12895 tablet_bootstrap.cc:492] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: No bootstrap required, opened a new log
I20260128 20:51:28.531560 12894 tablet_bootstrap.cc:492] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: No bootstrap required, opened a new log
I20260128 20:51:28.531409 12893 raft_consensus.cc:359] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.531675 12893 raft_consensus.cc:385] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:28.531737 12893 raft_consensus.cc:740] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcceb005c7c548679dd16848295150e7, State: Initialized, Role: FOLLOWER
I20260128 20:51:28.531834 12895 ts_tablet_manager.cc:1403] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:51:28.531842 12894 ts_tablet_manager.cc:1403] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 20:51:28.532294 12893 consensus_queue.cc:260] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.533398 12893 ts_tablet_manager.cc:1434] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260128 20:51:28.533632 12727 heartbeater.cc:499] Master 127.4.183.190:45085 was elected leader, sending a full tablet report...
I20260128 20:51:28.535266 12894 raft_consensus.cc:359] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.535532 12894 raft_consensus.cc:385] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:28.535596 12894 raft_consensus.cc:740] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e7ed4ed52848ce89ffdc349ae45ca5, State: Initialized, Role: FOLLOWER
I20260128 20:51:28.535640 12895 raft_consensus.cc:359] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.535944 12895 raft_consensus.cc:385] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:28.536149 12894 consensus_queue.cc:260] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.536013 12895 raft_consensus.cc:740] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353acb53f731489badbeb29923c2f9dc, State: Initialized, Role: FOLLOWER
I20260128 20:51:28.536842 12895 consensus_queue.cc:260] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.537335 12894 ts_tablet_manager.cc:1434] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260128 20:51:28.537817 12895 ts_tablet_manager.cc:1434] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260128 20:51:28.539287 12863 heartbeater.cc:499] Master 127.4.183.190:45085 was elected leader, sending a full tablet report...
I20260128 20:51:28.541280 12591 heartbeater.cc:499] Master 127.4.183.190:45085 was elected leader, sending a full tablet report...
W20260128 20:51:28.701246 12865 tablet.cc:2378] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:28.701247 12901 raft_consensus.cc:493] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:28.701470 12901 raft_consensus.cc:515] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.702832 12901 leader_election.cc:290] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173), fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:28.707176 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.703304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:45699 (local address 127.4.183.129:36029)
0128 20:51:28.703889 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.703907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.704011 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.704421 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.704424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.704455 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:28.704843 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.704854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.705845 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.705850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.706674 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.706684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.706820 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.706847 (+ 27us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.706885 (+ 38us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.706967 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":204,"threads_started":1}
I20260128 20:51:28.708079 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.703304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:40173 (local address 127.4.183.129:39733)
0128 20:51:28.704111 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:28.704130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:28.704269 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:28.704676 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:28.704679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:28.704712 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:28.705093 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.705102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.706243 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.706248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:28.707326 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:28.707346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.707642 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.707674 (+ 32us) client_negotiation.cc:769] Sending connection context
0128 20:51:28.707743 (+ 69us) client_negotiation.cc:240] Negotiation successful
0128 20:51:28.707864 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":370,"thread_start_us":101,"threads_started":1}
I20260128 20:51:28.708078 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.703602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:36029 (local address 127.4.183.130:45699)
0128 20:51:28.703950 (+ 348us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.703955 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.703979 (+ 24us) server_negotiation.cc:408] Connection header received
0128 20:51:28.704177 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.704184 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.704265 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.704356 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:28.705028 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.705703 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.706982 (+ 1279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.707766 (+ 784us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.707817 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.707896 (+ 79us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.707974 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
W20260128 20:51:28.708066 12729 tablet.cc:2378] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:28.708722 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.703373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:39733 (local address 127.4.183.131:40173)
0128 20:51:28.703538 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.703545 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.704258 (+ 713us) server_negotiation.cc:408] Connection header received
0128 20:51:28.704419 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.704423 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.704483 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.704603 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:28.705314 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.706030 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.707646 (+ 1616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.708371 (+ 725us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.708419 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.708490 (+ 71us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.708556 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:51:28.708900 12680 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182c4772e6a24daa8358b4d4bf35a53c" candidate_uuid: "353acb53f731489badbeb29923c2f9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcceb005c7c548679dd16848295150e7" is_pre_election: true
I20260128 20:51:28.709340 12680 raft_consensus.cc:2468] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353acb53f731489badbeb29923c2f9dc in term 0.
I20260128 20:51:28.709403 12816 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182c4772e6a24daa8358b4d4bf35a53c" candidate_uuid: "353acb53f731489badbeb29923c2f9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" is_pre_election: true
I20260128 20:51:28.709970 12477 leader_election.cc:304] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353acb53f731489badbeb29923c2f9dc, fcceb005c7c548679dd16848295150e7; no voters:
I20260128 20:51:28.710034 12816 raft_consensus.cc:2468] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353acb53f731489badbeb29923c2f9dc in term 0.
I20260128 20:51:28.710309 12901 raft_consensus.cc:2804] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:28.710402 12901 raft_consensus.cc:493] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:28.710433 12901 raft_consensus.cc:3060] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:28.711689 12901 raft_consensus.cc:515] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.712263 12901 leader_election.cc:290] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [CANDIDATE]: Term 1 election: Requested vote from peers 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173), fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:28.712815 12816 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182c4772e6a24daa8358b4d4bf35a53c" candidate_uuid: "353acb53f731489badbeb29923c2f9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5"
I20260128 20:51:28.713032 12816 raft_consensus.cc:3060] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:28.713270 12680 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182c4772e6a24daa8358b4d4bf35a53c" candidate_uuid: "353acb53f731489badbeb29923c2f9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcceb005c7c548679dd16848295150e7"
I20260128 20:51:28.713464 12680 raft_consensus.cc:3060] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:28.714287 12816 raft_consensus.cc:2468] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353acb53f731489badbeb29923c2f9dc in term 1.
I20260128 20:51:28.714634 12680 raft_consensus.cc:2468] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353acb53f731489badbeb29923c2f9dc in term 1.
I20260128 20:51:28.714876 12480 leader_election.cc:304] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e7ed4ed52848ce89ffdc349ae45ca5, 353acb53f731489badbeb29923c2f9dc; no voters:
I20260128 20:51:28.715097 12901 raft_consensus.cc:2804] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:28.715376 12901 raft_consensus.cc:697] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 1 LEADER]: Becoming Leader. State: Replica: 353acb53f731489badbeb29923c2f9dc, State: Running, Role: LEADER
I20260128 20:51:28.716178 12901 consensus_queue.cc:237] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } }
I20260128 20:51:28.720646 11595 catalog_manager.cc:5654] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 353acb53f731489badbeb29923c2f9dc (127.4.183.129). New cstate: current_term: 1 leader_uuid: "353acb53f731489badbeb29923c2f9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } health_report { overall_health: UNKNOWN } } }
W20260128 20:51:28.725364 12593 tablet.cc:2378] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:28.760839 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.755676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.4.183.129:46141)
0128 20:51:28.755826 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.755830 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.755846 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:51:28.756000 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.756003 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.756066 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.756176 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:28.756776 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.757372 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.758429 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.758691 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.758851 (+ 160us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:28.760189 (+ 1338us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:28.760368 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.760537 (+ 169us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.760648 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260128 20:51:28.768045 12816 raft_consensus.cc:1275] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Refusing update from remote peer 353acb53f731489badbeb29923c2f9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:28.768045 12680 raft_consensus.cc:1275] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Refusing update from remote peer 353acb53f731489badbeb29923c2f9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:28.769045 12901 consensus_queue.cc:1048] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:28.769384 12904 consensus_queue.cc:1048] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:28.779601 12907 mvcc.cc:204] Tried to move back new op lower bound from 7248418769984638976 to 7248418769784647680. Current Snapshot: MvccSnapshot[applied={T|T < 7248418769984638976}]
I20260128 20:51:28.780483 12908 mvcc.cc:204] Tried to move back new op lower bound from 7248418769984638976 to 7248418769784647680. Current Snapshot: MvccSnapshot[applied={T|T < 7248418769984638976}]
I20260128 20:51:28.796730 11594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:28.796936 11594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:28.803647 11594 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 55b783e56be54266bb6628d8f392e2ef: Sending DeleteTablet for 3 replicas of tablet 182c4772e6a24daa8358b4d4bf35a53c
I20260128 20:51:28.804697 12796 tablet_service.cc:1552] Processing DeleteTablet for tablet 182c4772e6a24daa8358b4d4bf35a53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:28 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:33552
I20260128 20:51:28.805608 12660 tablet_service.cc:1552] Processing DeleteTablet for tablet 182c4772e6a24daa8358b4d4bf35a53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:28 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:42826
I20260128 20:51:28.806008 12524 tablet_service.cc:1552] Processing DeleteTablet for tablet 182c4772e6a24daa8358b4d4bf35a53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:28 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:49050
I20260128 20:51:28.806566 12918 tablet_replica.cc:333] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: stopping tablet replica
I20260128 20:51:28.806960 12918 raft_consensus.cc:2243] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:28.807066 12919 tablet_replica.cc:333] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: stopping tablet replica
I20260128 20:51:28.807301 12919 raft_consensus.cc:2243] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:28.807559 12918 raft_consensus.cc:2272] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:28.807685 12919 raft_consensus.cc:2272] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:28.809303 12918 ts_tablet_manager.cc:1916] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:28.809303 12919 ts_tablet_manager.cc:1916] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:28.809556 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 11563
I20260128 20:51:28.812856 12919 ts_tablet_manager.cc:1929] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:28.813021 12919 log.cc:1199] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/182c4772e6a24daa8358b4d4bf35a53c
I20260128 20:51:28.813421 12919 ts_tablet_manager.cc:1950] T 182c4772e6a24daa8358b4d4bf35a53c P 353acb53f731489badbeb29923c2f9dc: Deleting consensus metadata
I20260128 20:51:28.813437 12918 ts_tablet_manager.cc:1929] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:28.813692 12918 log.cc:1199] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/182c4772e6a24daa8358b4d4bf35a53c
I20260128 20:51:28.814079 12918 ts_tablet_manager.cc:1950] T 182c4772e6a24daa8358b4d4bf35a53c P fcceb005c7c548679dd16848295150e7: Deleting consensus metadata
I20260128 20:51:28.815097 12917 tablet_replica.cc:333] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: stopping tablet replica
I20260128 20:51:28.815538 12917 raft_consensus.cc:2243] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:28.815907 12917 raft_consensus.cc:2272] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:28.817477 12917 ts_tablet_manager.cc:1916] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260128 20:51:28.818837 12477 connection.cc:537] server connection from 127.0.0.1:49050 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260128 20:51:28.818853 12615 connection.cc:537] server connection from 127.0.0.1:42826 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260128 20:51:28.820192 12917 ts_tablet_manager.cc:1929] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:28.820300 12917 log.cc:1199] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/182c4772e6a24daa8358b4d4bf35a53c
I20260128 20:51:28.821070 12917 ts_tablet_manager.cc:1950] T 182c4772e6a24daa8358b4d4bf35a53c P 27e7ed4ed52848ce89ffdc349ae45ca5: Deleting consensus metadata
W20260128 20:51:28.822481 12749 connection.cc:441] server connection from 127.0.0.1:33552 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33552 (request call id 1) could send its response
I20260128 20:51:28.831966 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.822928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.4.183.189:45875)
0128 20:51:28.823085 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.823092 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.823338 (+ 246us) server_negotiation.cc:408] Connection header received
0128 20:51:28.823512 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.823516 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.823577 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.823674 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.824595 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.825252 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.826089 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.826682 (+ 593us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.827615 (+ 933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.827690 (+ 75us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.827692 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.827722 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.829807 (+ 2085us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.830287 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.830293 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.830295 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.830355 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.830773 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.830780 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.830785 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.831037 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.831206 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.831501 (+ 295us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.831823 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260128 20:51:28.834029 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:28.823346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.4.183.188:42553)
0128 20:51:28.823482 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:28.823486 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:28.824256 (+ 770us) server_negotiation.cc:408] Connection header received
0128 20:51:28.824439 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:28.824443 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:28.824502 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:28.824559 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:28.826279 (+ 1720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.827120 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:28.828114 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:28.828368 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:28.829513 (+ 1145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:28.829538 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:28.829542 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:28.829578 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:28.831631 (+ 2053us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.832401 (+ 770us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.832405 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.832407 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.832465 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:28.832933 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:28.832936 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:28.832938 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:28.833119 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:28.833350 (+ 231us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:28.833714 (+ 364us) server_negotiation.cc:300] Negotiation successful
0128 20:51:28.833831 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260128 20:51:29.786819 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:29.786357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:59687)
0128 20:51:29.786644 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:29.786721 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
W20260128 20:51:29.787190 12863 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:45085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:29.790791 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:29.790172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:56325)
0128 20:51:29.790602 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:29.790692 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
W20260128 20:51:29.791252 12591 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:45085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:29.810628 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:29.810074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:45019)
0128 20:51:29.810402 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:29.810493 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
W20260128 20:51:29.811051 12727 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:45085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.303740 12930 raft_consensus.cc:493] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55b783e56be54266bb6628d8f392e2ef)
I20260128 20:51:30.303983 12930 raft_consensus.cc:515] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:30.304126 12931 raft_consensus.cc:493] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55b783e56be54266bb6628d8f392e2ef)
I20260128 20:51:30.304370 12931 raft_consensus.cc:515] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:30.305262 12930 leader_election.cc:290] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188:42553)
I20260128 20:51:30.305642 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "369959360abb4f4e89afcd777b02bbf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "28230a487eeb47ca815837c6f045f379" is_pre_election: true
I20260128 20:51:30.305799 12217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 369959360abb4f4e89afcd777b02bbf1 in term 1.
I20260128 20:51:30.306000 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.305401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55488)
0128 20:51:30.305736 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.305842 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":60,"threads_started":1}
I20260128 20:51:30.305970 12407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28230a487eeb47ca815837c6f045f379" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "369959360abb4f4e89afcd777b02bbf1" is_pre_election: true
I20260128 20:51:30.306103 12382 leader_election.cc:304] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28230a487eeb47ca815837c6f045f379, 369959360abb4f4e89afcd777b02bbf1; no voters:
W20260128 20:51:30.306173 12385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.306232 12407 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28230a487eeb47ca815837c6f045f379 in term 1.
I20260128 20:51:30.306388 12934 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260128 20:51:30.306454 12934 raft_consensus.cc:493] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 55b783e56be54266bb6628d8f392e2ef)
I20260128 20:51:30.306522 12934 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 1 FOLLOWER]: Advancing to term 2
I20260128 20:51:30.306602 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.305397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55504)
0128 20:51:30.305814 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.306198 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":85,"threads_started":1}
I20260128 20:51:30.306649 12195 leader_election.cc:304] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28230a487eeb47ca815837c6f045f379, 369959360abb4f4e89afcd777b02bbf1; no voters:
I20260128 20:51:30.306036 12931 leader_election.cc:290] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 28230a487eeb47ca815837c6f045f379 (127.4.183.189:45875)
W20260128 20:51:30.306898 12195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.307056 12930 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260128 20:51:30.307119 12930 raft_consensus.cc:493] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 55b783e56be54266bb6628d8f392e2ef)
I20260128 20:51:30.307161 12930 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 1 FOLLOWER]: Advancing to term 2
I20260128 20:51:30.307163 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.306905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55516)
0128 20:51:30.307054 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.307099 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260128 20:51:30.307271 12385 leader_election.cc:336] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.307776 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.307503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55524)
0128 20:51:30.307653 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.307705 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260128 20:51:30.307952 12195 leader_election.cc:336] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.308002 12934 raft_consensus.cc:515] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:30.308440 12934 leader_election.cc:290] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 2 election: Requested vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 28230a487eeb47ca815837c6f045f379 (127.4.183.189:45875)
I20260128 20:51:30.308401 12930 raft_consensus.cc:515] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:30.308794 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.308523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55534)
0128 20:51:30.308650 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.308698 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260128 20:51:30.308801 12930 leader_election.cc:290] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [CANDIDATE]: Term 2 election: Requested vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 369959360abb4f4e89afcd777b02bbf1 (127.4.183.188:42553)
I20260128 20:51:30.308849 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "369959360abb4f4e89afcd777b02bbf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "28230a487eeb47ca815837c6f045f379"
I20260128 20:51:30.309029 12217 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 369959360abb4f4e89afcd777b02bbf1 in current term 2: Already voted for candidate 28230a487eeb47ca815837c6f045f379 in this term.
I20260128 20:51:30.309234 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.308922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55540)
0128 20:51:30.309055 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.309103 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260128 20:51:30.309414 12407 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28230a487eeb47ca815837c6f045f379" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "369959360abb4f4e89afcd777b02bbf1"
I20260128 20:51:30.309592 12407 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 28230a487eeb47ca815837c6f045f379 in current term 2: Already voted for candidate 369959360abb4f4e89afcd777b02bbf1 in this term.
I20260128 20:51:30.309597 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.309331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55548)
0128 20:51:30.309471 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.309521 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260128 20:51:30.309778 12385 leader_election.cc:336] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.309860 12385 leader_election.cc:304] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 369959360abb4f4e89afcd777b02bbf1; no voters: 28230a487eeb47ca815837c6f045f379, 55b783e56be54266bb6628d8f392e2ef
I20260128 20:51:30.310021 12934 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260128 20:51:30.310123 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.309864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55558)
0128 20:51:30.310004 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.310057 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260128 20:51:30.310222 12195 leader_election.cc:336] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:30.310319 12195 leader_election.cc:304] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28230a487eeb47ca815837c6f045f379; no voters: 369959360abb4f4e89afcd777b02bbf1, 55b783e56be54266bb6628d8f392e2ef
I20260128 20:51:30.310451 12930 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260128 20:51:30.788373 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.787830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:51137)
0128 20:51:30.788196 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.788271 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260128 20:51:30.792243 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.791810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:36995)
0128 20:51:30.792065 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.792138 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260128 20:51:30.812566 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:30.812075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:36585)
0128 20:51:30.812390 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:30.812459 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260128 20:51:31.789723 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:31.789216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:41457)
0128 20:51:31.789538 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:31.789610 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
I20260128 20:51:31.793431 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:31.792935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:36451)
0128 20:51:31.793219 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:31.793307 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260128 20:51:31.814713 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:31.813993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:48199)
0128 20:51:31.814478 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:31.814590 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":145,"threads_started":1}
I20260128 20:51:31.998389 12943 raft_consensus.cc:493] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:31.998549 12943 raft_consensus.cc:515] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:31.999027 12943 leader_election.cc:290] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 28230a487eeb47ca815837c6f045f379 (127.4.183.189:45875)
I20260128 20:51:31.999650 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:31.999142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55794)
0128 20:51:31.999452 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:31.999562 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260128 20:51:31.999639 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "369959360abb4f4e89afcd777b02bbf1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "28230a487eeb47ca815837c6f045f379" is_pre_election: true
I20260128 20:51:31.999825 12217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 369959360abb4f4e89afcd777b02bbf1 in term 2.
I20260128 20:51:32.000334 12382 leader_election.cc:304] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28230a487eeb47ca815837c6f045f379, 369959360abb4f4e89afcd777b02bbf1; no voters:
I20260128 20:51:32.000655 12943 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260128 20:51:32.000715 12943 raft_consensus.cc:493] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:32.000901 12943 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 2 FOLLOWER]: Advancing to term 3
I20260128 20:51:32.001021 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.000422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55802)
0128 20:51:32.000792 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.000857 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
W20260128 20:51:32.001214 12385 leader_election.cc:336] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:32.002228 12943 raft_consensus.cc:515] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:32.002660 12943 leader_election.cc:290] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 3 election: Requested vote from peers 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085), 28230a487eeb47ca815837c6f045f379 (127.4.183.189:45875)
I20260128 20:51:32.003214 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.002922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55814)
0128 20:51:32.003095 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.003146 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260128 20:51:32.003317 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "369959360abb4f4e89afcd777b02bbf1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "28230a487eeb47ca815837c6f045f379"
I20260128 20:51:32.003487 12217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 2 FOLLOWER]: Advancing to term 3
I20260128 20:51:32.003969 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.003735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55818)
0128 20:51:32.003861 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.003912 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260128 20:51:32.004079 12385 leader_election.cc:336] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
I20260128 20:51:32.004621 12217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 369959360abb4f4e89afcd777b02bbf1 in term 3.
I20260128 20:51:32.005035 12382 leader_election.cc:304] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28230a487eeb47ca815837c6f045f379, 369959360abb4f4e89afcd777b02bbf1; no voters: 55b783e56be54266bb6628d8f392e2ef
I20260128 20:51:32.005291 12943 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 3 FOLLOWER]: Leader election won for term 3
I20260128 20:51:32.005497 12943 raft_consensus.cc:697] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [term 3 LEADER]: Becoming Leader. State: Replica: 369959360abb4f4e89afcd777b02bbf1, State: Running, Role: LEADER
I20260128 20:51:32.005784 12943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } }
I20260128 20:51:32.007103 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369959360abb4f4e89afcd777b02bbf1. Latest consensus state: current_term: 3 leader_uuid: "369959360abb4f4e89afcd777b02bbf1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:32.007246 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:32.007563 12948 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:32.010057 12948 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:32.010565 12948 catalog_manager.cc:1269] Loaded cluster ID: 4fbd71821df24198bcc7e6c817ea9858
I20260128 20:51:32.010627 12948 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:32.011230 12948 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:32.011682 12948 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Loaded TSK: 0
I20260128 20:51:32.012441 12948 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:51:32.079545 12397 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:32.082288 12397 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:32.085681 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.085384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55836)
0128 20:51:32.085518 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.085595 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 20:51:32.085829 12217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [term 3 FOLLOWER]: Refusing update from remote peer 369959360abb4f4e89afcd777b02bbf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260128 20:51:32.086573 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.086221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.0.0.1:55848)
0128 20:51:32.086337 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.086393 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260128 20:51:32.086378 12946 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260128 20:51:32.086740 12385 consensus_peers.cc:597] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 -> Peer 55b783e56be54266bb6628d8f392e2ef (127.4.183.190:45085): Couldn't send request to peer 55b783e56be54266bb6628d8f392e2ef. Status: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260128 20:51:32.088274 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369959360abb4f4e89afcd777b02bbf1. Latest consensus state: current_term: 3 leader_uuid: "369959360abb4f4e89afcd777b02bbf1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:32.088419 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:32.089845 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "369959360abb4f4e89afcd777b02bbf1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:32.089986 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:32.090240 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "369959360abb4f4e89afcd777b02bbf1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "55b783e56be54266bb6628d8f392e2ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 45085 } } peers { permanent_uuid: "28230a487eeb47ca815837c6f045f379" member_type: VOTER last_known_addr { host: "127.4.183.189" port: 45875 } attrs { promote: false } } peers { permanent_uuid: "369959360abb4f4e89afcd777b02bbf1" member_type: VOTER last_known_addr { host: "127.4.183.188" port: 42553 } attrs { promote: false } } }
I20260128 20:51:32.090520 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28230a487eeb47ca815837c6f045f379 [sys.catalog]: This master's current role is: FOLLOWER
I20260128 20:51:32.101127 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.097745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:46141 (local address 127.0.0.1:42454)
0128 20:51:32.097897 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.097912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:32.098036 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:32.098555 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:32.098558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:32.098579 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:32.098893 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.098906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.099919 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.099922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:32.100845 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.100853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.100917 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.100936 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:51:32.100966 (+ 30us) client_negotiation.cc:240] Negotiation successful
0128 20:51:32.101019 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260128 20:51:32.102140 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.098592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:40173 (local address 127.0.0.1:57630)
0128 20:51:32.098855 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.098867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:32.098946 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:32.099530 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:32.099533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:32.099553 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:32.099778 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.099785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.100820 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.100824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:32.101742 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.101752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.101867 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.101888 (+ 21us) client_negotiation.cc:769] Sending connection context
0128 20:51:32.101942 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 20:51:32.102013 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260128 20:51:32.102283 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.097886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.4.183.129:46141)
0128 20:51:32.098263 (+ 377us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:32.098268 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:32.098282 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:32.098335 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:32.098339 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:32.098395 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:32.099013 (+ 618us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:32.099123 (+ 110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.099773 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.101559 (+ 1786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.101989 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.102027 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:32.102080 (+ 53us) server_negotiation.cc:300] Negotiation successful
0128 20:51:32.102149 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260128 20:51:32.102286 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.098201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.4.183.130:45699)
0128 20:51:32.098453 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:32.098460 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:32.098783 (+ 323us) server_negotiation.cc:408] Connection header received
0128 20:51:32.098844 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:32.098847 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:32.098900 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:32.099044 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:32.099450 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.100170 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.101320 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.101809 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.101865 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:32.102057 (+ 192us) server_negotiation.cc:300] Negotiation successful
0128 20:51:32.102134 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260128 20:51:32.102679 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.098108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:45699 (local address 127.0.0.1:56660)
0128 20:51:32.098579 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.098597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:32.098695 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:32.099054 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:32.099057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:32.099075 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:32.099308 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.099316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.100308 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.100313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:32.101177 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.101186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.101915 (+ 729us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.101935 (+ 20us) client_negotiation.cc:769] Sending connection context
0128 20:51:32.102490 (+ 555us) client_negotiation.cc:240] Negotiation successful
0128 20:51:32.102568 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":102,"threads_started":1}
I20260128 20:51:32.103302 12524 tablet_service.cc:1505] Processing CreateTablet for tablet 8740826edfd64623bbdc13287740385d (DEFAULT_TABLE table=test-table [id=90ee3b71fb5b47ca8028298f53e1ae99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:32.103302 12660 tablet_service.cc:1505] Processing CreateTablet for tablet 8740826edfd64623bbdc13287740385d (DEFAULT_TABLE table=test-table [id=90ee3b71fb5b47ca8028298f53e1ae99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:32.103626 12660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8740826edfd64623bbdc13287740385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:32.103626 12524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8740826edfd64623bbdc13287740385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:32.105710 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.098689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.4.183.131:40173)
0128 20:51:32.099260 (+ 571us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:32.099265 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:32.099277 (+ 12us) server_negotiation.cc:408] Connection header received
0128 20:51:32.099327 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:32.099330 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:32.099403 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:32.099493 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:32.099916 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.100690 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.101900 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.105422 (+ 3522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.105467 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:32.105528 (+ 61us) server_negotiation.cc:300] Negotiation successful
0128 20:51:32.105591 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":97,"threads_started":1}
I20260128 20:51:32.106374 12796 tablet_service.cc:1505] Processing CreateTablet for tablet 8740826edfd64623bbdc13287740385d (DEFAULT_TABLE table=test-table [id=90ee3b71fb5b47ca8028298f53e1ae99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:32.106668 12960 tablet_bootstrap.cc:492] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc: Bootstrap starting.
I20260128 20:51:32.106688 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8740826edfd64623bbdc13287740385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:32.107637 12960 tablet_bootstrap.cc:654] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:32.108405 12961 tablet_bootstrap.cc:492] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7: Bootstrap starting.
I20260128 20:51:32.109236 12963 tablet_bootstrap.cc:492] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Bootstrap starting.
I20260128 20:51:32.109380 12961 tablet_bootstrap.cc:654] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:32.110244 12963 tablet_bootstrap.cc:654] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:32.110563 12960 tablet_bootstrap.cc:492] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc: No bootstrap required, opened a new log
I20260128 20:51:32.110695 12960 ts_tablet_manager.cc:1403] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260128 20:51:32.111166 12960 raft_consensus.cc:359] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.111274 12960 raft_consensus.cc:385] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:32.111296 12960 raft_consensus.cc:740] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353acb53f731489badbeb29923c2f9dc, State: Initialized, Role: FOLLOWER
I20260128 20:51:32.111537 12963 tablet_bootstrap.cc:492] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: No bootstrap required, opened a new log
I20260128 20:51:32.111655 12963 ts_tablet_manager.cc:1403] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 20:51:32.111931 12961 tablet_bootstrap.cc:492] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7: No bootstrap required, opened a new log
I20260128 20:51:32.112037 12961 ts_tablet_manager.cc:1403] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260128 20:51:32.112131 12963 raft_consensus.cc:359] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.112267 12963 raft_consensus.cc:385] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:32.112300 12963 raft_consensus.cc:740] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e7ed4ed52848ce89ffdc349ae45ca5, State: Initialized, Role: FOLLOWER
I20260128 20:51:32.112485 12963 consensus_queue.cc:260] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.112543 12961 raft_consensus.cc:359] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.112701 12961 raft_consensus.cc:385] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:32.112771 12961 raft_consensus.cc:740] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcceb005c7c548679dd16848295150e7, State: Initialized, Role: FOLLOWER
I20260128 20:51:32.112978 12963 ts_tablet_manager.cc:1434] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260128 20:51:32.113001 12960 consensus_queue.cc:260] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.113214 12961 consensus_queue.cc:260] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.113647 12961 ts_tablet_manager.cc:1434] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260128 20:51:32.114056 12960 ts_tablet_manager.cc:1434] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:32.114130 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.113793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:42449)
0128 20:51:32.113983 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.114039 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260128 20:51:32.115504 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.115241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:58197)
0128 20:51:32.115360 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.115414 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 20:51:32.116225 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.115971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:46401)
0128 20:51:32.116094 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.116146 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 20:51:32.117733 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.117462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:32953)
0128 20:51:32.117609 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.117661 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260128 20:51:32.118395 12590 heartbeater.cc:499] Master 127.4.183.188:42553 was elected leader, sending a full tablet report...
I20260128 20:51:32.118446 12862 heartbeater.cc:499] Master 127.4.183.188:42553 was elected leader, sending a full tablet report...
I20260128 20:51:32.362190 12966 raft_consensus.cc:493] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:32.362339 12966 raft_consensus.cc:515] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.363550 12966 leader_election.cc:290] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141), fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:32.367369 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.363819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:46141 (local address 127.4.183.131:51233)
0128 20:51:32.363946 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.363965 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:32.364089 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:32.364451 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:32.364455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:32.364479 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:32.364796 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.364808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.366193 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.366197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:32.367018 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.367027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.367133 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.367154 (+ 21us) client_negotiation.cc:769] Sending connection context
0128 20:51:32.367193 (+ 39us) client_negotiation.cc:240] Negotiation successful
0128 20:51:32.367252 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260128 20:51:32.367983 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.363964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:51233 (local address 127.4.183.129:46141)
0128 20:51:32.364123 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:32.364128 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:32.364181 (+ 53us) server_negotiation.cc:408] Connection header received
0128 20:51:32.364251 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:32.364274 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:32.364339 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:32.364425 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:32.365020 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.366070 (+ 1050us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.367174 (+ 1104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.367689 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.367734 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:32.367787 (+ 53us) server_negotiation.cc:300] Negotiation successful
0128 20:51:32.367866 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 20:51:32.368599 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.364097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:45699 (local address 127.4.183.131:46801)
0128 20:51:32.364487 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.364503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:32.364602 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:32.365016 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:32.365022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:32.365074 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:32.365658 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.365674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.366723 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.366727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:32.367474 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:32.367484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.368203 (+ 719us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.368226 (+ 23us) client_negotiation.cc:769] Sending connection context
0128 20:51:32.368339 (+ 113us) client_negotiation.cc:240] Negotiation successful
0128 20:51:32.368409 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":173,"threads_started":1}
I20260128 20:51:32.368577 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.364286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:46801 (local address 127.4.183.130:45699)
0128 20:51:32.364451 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:32.364456 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:32.364599 (+ 143us) server_negotiation.cc:408] Connection header received
0128 20:51:32.364688 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:32.364692 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:32.364741 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:32.364831 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:32.365885 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.366527 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.367660 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.368158 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.368196 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:32.368370 (+ 174us) server_negotiation.cc:300] Negotiation successful
0128 20:51:32.368442 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260128 20:51:32.368633 12544 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8740826edfd64623bbdc13287740385d" candidate_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353acb53f731489badbeb29923c2f9dc" is_pre_election: true
I20260128 20:51:32.368912 12544 raft_consensus.cc:2468] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27e7ed4ed52848ce89ffdc349ae45ca5 in term 0.
I20260128 20:51:32.369290 12680 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8740826edfd64623bbdc13287740385d" candidate_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcceb005c7c548679dd16848295150e7" is_pre_election: true
I20260128 20:51:32.369401 12749 leader_election.cc:304] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e7ed4ed52848ce89ffdc349ae45ca5, 353acb53f731489badbeb29923c2f9dc; no voters:
I20260128 20:51:32.369464 12680 raft_consensus.cc:2468] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27e7ed4ed52848ce89ffdc349ae45ca5 in term 0.
I20260128 20:51:32.369747 12966 raft_consensus.cc:2804] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:32.369827 12966 raft_consensus.cc:493] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:32.369897 12966 raft_consensus.cc:3060] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:32.371043 12966 raft_consensus.cc:515] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.371471 12966 leader_election.cc:290] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [CANDIDATE]: Term 1 election: Requested vote from peers 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141), fcceb005c7c548679dd16848295150e7 (127.4.183.130:45699)
I20260128 20:51:32.372125 12544 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8740826edfd64623bbdc13287740385d" candidate_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353acb53f731489badbeb29923c2f9dc"
I20260128 20:51:32.372144 12680 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8740826edfd64623bbdc13287740385d" candidate_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcceb005c7c548679dd16848295150e7"
I20260128 20:51:32.372263 12544 raft_consensus.cc:3060] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:32.372289 12680 raft_consensus.cc:3060] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:32.373347 12680 raft_consensus.cc:2468] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27e7ed4ed52848ce89ffdc349ae45ca5 in term 1.
I20260128 20:51:32.373414 12544 raft_consensus.cc:2468] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27e7ed4ed52848ce89ffdc349ae45ca5 in term 1.
I20260128 20:51:32.373840 12749 leader_election.cc:304] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e7ed4ed52848ce89ffdc349ae45ca5, fcceb005c7c548679dd16848295150e7; no voters:
I20260128 20:51:32.374125 12966 raft_consensus.cc:2804] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:32.374408 12966 raft_consensus.cc:697] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 27e7ed4ed52848ce89ffdc349ae45ca5, State: Running, Role: LEADER
I20260128 20:51:32.374997 12966 consensus_queue.cc:237] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } }
I20260128 20:51:32.375603 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.375143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:57121)
0128 20:51:32.375445 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.375512 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260128 20:51:32.378327 12397 catalog_manager.cc:5654] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131). New cstate: current_term: 1 leader_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27e7ed4ed52848ce89ffdc349ae45ca5" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 40173 } health_report { overall_health: HEALTHY } } }
I20260128 20:51:32.401957 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.398139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.4.183.131:40173)
0128 20:51:32.398278 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:32.398282 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:32.398296 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:32.398440 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:32.398443 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:32.398502 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:32.398591 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:32.398958 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.399475 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:32.400200 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:32.400359 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:32.400418 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:32.401351 (+ 933us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:32.401629 (+ 278us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:32.401672 (+ 43us) server_negotiation.cc:300] Negotiation successful
0128 20:51:32.401734 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260128 20:51:32.405323 12680 raft_consensus.cc:1275] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Refusing update from remote peer 27e7ed4ed52848ce89ffdc349ae45ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:32.405560 12544 raft_consensus.cc:1275] T 8740826edfd64623bbdc13287740385d P 353acb53f731489badbeb29923c2f9dc [term 1 FOLLOWER]: Refusing update from remote peer 27e7ed4ed52848ce89ffdc349ae45ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:32.405927 12966 consensus_queue.cc:1048] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcceb005c7c548679dd16848295150e7" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:32.406164 12970 consensus_queue.cc:1048] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353acb53f731489badbeb29923c2f9dc" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 46141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:32.407248 12726 heartbeater.cc:499] Master 127.4.183.188:42553 was elected leader, sending a full tablet report...
I20260128 20:51:32.408385 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.408037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:58319)
0128 20:51:32.408224 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.408290 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260128 20:51:32.408900 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.408570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:41305)
0128 20:51:32.408728 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.408824 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260128 20:51:32.411986 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.411576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:54535)
0128 20:51:32.411831 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.411892 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260128 20:51:32.413089 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.412758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.130:33243)
0128 20:51:32.412912 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.412998 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260128 20:51:32.413578 12974 mvcc.cc:204] Tried to move back new op lower bound from 7248418784885805056 to 7248418784771674112. Current Snapshot: MvccSnapshot[applied={T|T < 7248418784885805056 or (T in {7248418784885805056})}]
I20260128 20:51:32.416747 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.416486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.131:55697)
0128 20:51:32.416598 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.416655 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260128 20:51:32.417385 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.416919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:45085 (local address 127.4.183.129:34483)
0128 20:51:32.417229 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.417306 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260128 20:51:32.422024 12397 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:32.422345 12397 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:32.427238 12397 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 369959360abb4f4e89afcd777b02bbf1: Sending DeleteTablet for 3 replicas of tablet 8740826edfd64623bbdc13287740385d
I20260128 20:51:32.428145 12660 tablet_service.cc:1552] Processing DeleteTablet for tablet 8740826edfd64623bbdc13287740385d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:32 UTC) from {username='oryx', principal='oryx/127.4.183.188@KRBTEST.COM'} at 127.0.0.1:56660
I20260128 20:51:32.428185 12524 tablet_service.cc:1552] Processing DeleteTablet for tablet 8740826edfd64623bbdc13287740385d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:32 UTC) from {username='oryx', principal='oryx/127.4.183.188@KRBTEST.COM'} at 127.0.0.1:42454
I20260128 20:51:32.428930 12796 tablet_service.cc:1552] Processing DeleteTablet for tablet 8740826edfd64623bbdc13287740385d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:32 UTC) from {username='oryx', principal='oryx/127.4.183.188@KRBTEST.COM'} at 127.0.0.1:57630
I20260128 20:51:32.429390 12986 tablet_replica.cc:333] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: stopping tablet replica
I20260128 20:51:32.429581 12986 raft_consensus.cc:2243] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:32.429698 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 12461
I20260128 20:51:32.429992 12986 raft_consensus.cc:2272] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:32.431517 12986 ts_tablet_manager.cc:1916] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:32.434410 12986 ts_tablet_manager.cc:1929] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:32.434526 12986 log.cc:1199] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/8740826edfd64623bbdc13287740385d
I20260128 20:51:32.434837 12986 ts_tablet_manager.cc:1950] T 8740826edfd64623bbdc13287740385d P 27e7ed4ed52848ce89ffdc349ae45ca5: Deleting consensus metadata
I20260128 20:51:32.435853 12385 catalog_manager.cc:4985] TS 27e7ed4ed52848ce89ffdc349ae45ca5 (127.4.183.131:40173): tablet 8740826edfd64623bbdc13287740385d (table test-table [id=90ee3b71fb5b47ca8028298f53e1ae99]) successfully deleted
I20260128 20:51:32.439069 12984 tablet_replica.cc:333] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7: stopping tablet replica
W20260128 20:51:32.439826 12956 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:32.439320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:46141 (local address 127.0.0.1:42462)
0128 20:51:32.439458 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.439477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:32.439582 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:32.439690 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.129:46141: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
I20260128 20:51:32.440011 12984 raft_consensus.cc:2243] T 8740826edfd64623bbdc13287740385d P fcceb005c7c548679dd16848295150e7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260128 20:51:32.440038 12382 catalog_manager.cc:4712] TS 353acb53f731489badbeb29923c2f9dc (127.4.183.129:46141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8740826edfd64623bbdc13287740385d: Network error: Client connection negotiation failed: client connection to 127.4.183.129:46141: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260128 20:51:32.440056 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 12597
I20260128 20:51:32.447314 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 12733
I20260128 20:51:32.447656 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:32.447323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:45699 (local address 127.0.0.1:56670)
0128 20:51:32.447475 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:32.447540 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.130:45699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260128 20:51:32.454775 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 12067
I20260128 20:51:32.462541 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 12180
2026-01-28T20:51:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10053 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:51:32 dist-test-slave-hss7 krb5kdc[12989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:51:32 dist-test-slave-hss7 krb5kdc[12989](info): set up 2 sockets
Jan 28 20:51:32 dist-test-slave-hss7 krb5kdc[12989](info): commencing operation
krb5kdc: starting...
W20260128 20:51:34.531328 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.042s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:51:34 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:51:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:34.690649 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:34393
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:34393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:34.817943 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:34.818228 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:34.818288 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:34.822165 13005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:34.822268 13005 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260128 20:51:34.822289 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:34.822311 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:34.822330 13005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:34.822347 13005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:34.827813 13005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:34393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:34.829239 13005 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:34.830605 13005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:34.838388 13010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:34.838418 13013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:34.838388 13011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:34.838778 13005 server_base.cc:1060] running on GCE node
I20260128 20:51:34.839465 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:34.841275 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:34.842478 13005 hybrid_clock.cc:648] HybridClock initialized: now 1769633494842441 us; error 61 us; skew 500 ppm
Jan 28 20:51:34 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633494, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:34.845642 13005 init.cc:377] Logged in from keytab as oryx/127.4.183.190@KRBTEST.COM (short username oryx)
I20260128 20:51:34.846938 13005 webserver.cc:492] Webserver started at http://127.4.183.190:45831/ using document root <none> and password file <none>
I20260128 20:51:34.847575 13005 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:34.847658 13005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:34.847898 13005 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:34.849983 13005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d"
format_stamp: "Formatted at 2026-01-28 20:51:34 on dist-test-slave-hss7"
server_key: "25a9fae457ba3bff40eaa3048130f2d5"
server_key_iv: "ab153931408fb42a2c5902d50ed6747e"
server_key_version: "encryptionkey@0"
I20260128 20:51:34.850569 13005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d"
format_stamp: "Formatted at 2026-01-28 20:51:34 on dist-test-slave-hss7"
server_key: "25a9fae457ba3bff40eaa3048130f2d5"
server_key_iv: "ab153931408fb42a2c5902d50ed6747e"
server_key_version: "encryptionkey@0"
I20260128 20:51:34.855072 13005 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:34.858186 13020 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:34.859475 13005 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:51:34.859683 13005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d"
format_stamp: "Formatted at 2026-01-28 20:51:34 on dist-test-slave-hss7"
server_key: "25a9fae457ba3bff40eaa3048130f2d5"
server_key_iv: "ab153931408fb42a2c5902d50ed6747e"
server_key_version: "encryptionkey@0"
I20260128 20:51:34.859875 13005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:34.874063 13005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:34.877806 13005 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:34.878057 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:34.888007 13005 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:34393
I20260128 20:51:34.888042 13082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:34393 every 8 connection(s)
I20260128 20:51:34.889333 13005 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:34.892637 13083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:34.898752 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13005
I20260128 20:51:34.898948 13083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Bootstrap starting.
I20260128 20:51:34.899034 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:34.899302 4830 external_mini_cluster.cc:1468] Setting key 0f83d0ce7d9011d56ac0892eab1ad8ff
I20260128 20:51:34.901822 13083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:34.902748 13083 log.cc:826] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:34.905345 13083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: No bootstrap required, opened a new log
Jan 28 20:51:34 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:34.909614 13083 raft_consensus.cc:359] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 34393 } }
I20260128 20:51:34.909929 13083 raft_consensus.cc:385] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:34.910032 13083 raft_consensus.cc:740] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c6f7178eb04cacb9ef0a1ef6fa009d, State: Initialized, Role: FOLLOWER
I20260128 20:51:34.910622 13083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 34393 } }
I20260128 20:51:34.910815 13083 raft_consensus.cc:399] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:34.910904 13083 raft_consensus.cc:493] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:34.911036 13083 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:34.912082 13083 raft_consensus.cc:515] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 34393 } }
I20260128 20:51:34.912505 13083 leader_election.cc:304] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60c6f7178eb04cacb9ef0a1ef6fa009d; no voters:
I20260128 20:51:34.912930 13083 leader_election.cc:290] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:34.913033 13088 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:34.913357 13088 raft_consensus.cc:697] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [term 1 LEADER]: Becoming Leader. State: Replica: 60c6f7178eb04cacb9ef0a1ef6fa009d, State: Running, Role: LEADER
I20260128 20:51:34.913771 13088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 34393 } }
I20260128 20:51:34.914244 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:34.900730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.4.183.190:34393)
0128 20:51:34.901239 (+ 509us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:34.901249 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:34.901288 (+ 39us) server_negotiation.cc:408] Connection header received
0128 20:51:34.901931 (+ 643us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:34.901951 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:34.902336 (+ 385us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:34.902658 (+ 322us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:34.903790 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:34.904558 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:34.905635 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:34.905891 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:34.908857 (+ 2966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:34.908885 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:34.908899 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:34.908956 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:34.911130 (+ 2174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:34.911674 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:34.911681 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:34.911687 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:34.911791 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:34.912076 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:34.912079 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:34.912081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:34.912540 (+ 459us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:34.912815 (+ 275us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:34.913081 (+ 266us) server_negotiation.cc:300] Negotiation successful
0128 20:51:34.913480 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":168,"threads_started":1}
I20260128 20:51:34.914497 13083 sys_catalog.cc:565] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:34.915194 13088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 34393 } } }
I20260128 20:51:34.915334 13088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:34.915565 13088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60c6f7178eb04cacb9ef0a1ef6fa009d. Latest consensus state: current_term: 1 leader_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c6f7178eb04cacb9ef0a1ef6fa009d" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 34393 } } }
I20260128 20:51:34.915671 13088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:34.917488 13093 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:34.920580 13093 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:34.927588 13093 catalog_manager.cc:1357] Generated new cluster ID: f28e3e77c6ad49f5b8de2e81f3e8d288
I20260128 20:51:34.927735 13093 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:34.936072 13093 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:34.937501 13093 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:34.951125 13093 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Generated new TSK 0
I20260128 20:51:34.952055 13093 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:35.017416 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:34393
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:35.159510 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:35.159807 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:35.159873 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:35.163553 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:35.163664 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:35.163723 13111 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:51:35.163765 13111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:35.168718 13111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:34393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:35.170138 13111 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:35.171068 13111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:35.179756 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:35.179796 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:35.179774 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:35.180114 13111 server_base.cc:1060] running on GCE node
I20260128 20:51:35.180641 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:35.181550 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:35.182814 13111 hybrid_clock.cc:648] HybridClock initialized: now 1769633495182779 us; error 79 us; skew 500 ppm
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:35.186388 13111 init.cc:377] Logged in from keytab as oryx/127.4.183.129@KRBTEST.COM (short username oryx)
I20260128 20:51:35.188254 13111 webserver.cc:492] Webserver started at http://127.4.183.129:34711/ using document root <none> and password file <none>
I20260128 20:51:35.189021 13111 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:35.189116 13111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:35.189414 13111 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:35.191358 13111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "7f6e83fc466c4e169f49e61dda87d042"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "6fd26614f2b23ce99c117b0fa9d2a86f"
server_key_iv: "93c5ea137b0d65e192f194a3de5ce0d3"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.191992 13111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "7f6e83fc466c4e169f49e61dda87d042"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "6fd26614f2b23ce99c117b0fa9d2a86f"
server_key_iv: "93c5ea137b0d65e192f194a3de5ce0d3"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.196323 13111 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260128 20:51:35.199322 13126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:35.200662 13111 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:35.200847 13111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "7f6e83fc466c4e169f49e61dda87d042"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "6fd26614f2b23ce99c117b0fa9d2a86f"
server_key_iv: "93c5ea137b0d65e192f194a3de5ce0d3"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.200990 13111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:35.218967 13111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:35.223241 13111 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:35.223486 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:35.225085 13111 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:35.225194 13111 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:35.225291 13111 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:35.225340 13111 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:35.238454 13111 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:39527
I20260128 20:51:35.238669 13241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:39527 every 8 connection(s)
I20260128 20:51:35.240145 13111 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:35.240293 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.225995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:39777 (local address 127.4.183.190:34393)
0128 20:51:35.226156 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.226161 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.226965 (+ 804us) server_negotiation.cc:408] Connection header received
0128 20:51:35.227895 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.227902 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.227975 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.228085 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.229786 (+ 1701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.230683 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.231540 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.231749 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.234363 (+ 2614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.234390 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.234392 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.234423 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.236568 (+ 2145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.237665 (+ 1097us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.237670 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.237672 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.237746 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.238182 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.238186 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.238187 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.238481 (+ 294us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.238848 (+ 367us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.239818 (+ 970us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.240092 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 20:51:35.241031 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.226325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.4.183.129:39777)
0128 20:51:35.226788 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.226834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.227689 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.228277 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.228311 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.228897 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:35.229601 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.229615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.230865 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.230869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.231388 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.231397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.231686 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.232424 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:35.232451 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:35.234168 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:35.236761 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.236769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.236785 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.237459 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.237868 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.237872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.237876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.238054 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.238955 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:35.238963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:35.239375 (+ 412us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.239706 (+ 331us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.240139 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":140,"threads_started":1}
I20260128 20:51:35.242921 13242 heartbeater.cc:344] Connected to a master server at 127.4.183.190:34393
I20260128 20:51:35.243316 13242 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:35.244673 13242 heartbeater.cc:507] Master 127.4.183.190:34393 requested a full tablet report, sending...
I20260128 20:51:35.246740 13036 ts_manager.cc:194] Registered new tserver with Master: 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527)
I20260128 20:51:35.248001 13036 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:39777
I20260128 20:51:35.248457 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13111
I20260128 20:51:35.248610 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:35.248929 4830 external_mini_cluster.cc:1468] Setting key 45f84c3ed89816c3b63b512583f88245
I20260128 20:51:35.257411 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.246327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:57949 (local address 127.4.183.190:34393)
0128 20:51:35.246483 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.246488 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.249482 (+ 2994us) server_negotiation.cc:408] Connection header received
0128 20:51:35.249699 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.249702 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.249756 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.249840 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.250655 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.251405 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.252055 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.252300 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.253759 (+ 1459us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.253780 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.253783 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.253813 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.255480 (+ 1667us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.256105 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.256108 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.256109 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.256159 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.256462 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.256464 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.256465 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.256648 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.256731 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.257092 (+ 361us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.257283 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260128 20:51:35.257527 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.246209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.4.183.129:57949)
0128 20:51:35.249362 (+ 3153us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.249382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.249550 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.249957 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.249960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.250193 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:35.250478 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.250487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.251537 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.251540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.251933 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.251940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.252041 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.252661 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:35.252681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:35.253552 (+ 871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:35.255648 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.255656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.255661 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.255982 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.256269 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.256275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.256279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.256353 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.256757 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:35.256762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:35.256905 (+ 143us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.257041 (+ 136us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.257259 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3047,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for oryx/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:35.309367 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:34393
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:35.432183 13253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:35.432521 13253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:35.432619 13253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:35.436852 13253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:35.437049 13253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:35.437258 13253 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:51:35.437371 13253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:35.443161 13253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:34393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:35.444586 13253 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:35.445582 13253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:35.452845 13259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:35.452870 13258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:35.452847 13261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:35.453217 13253 server_base.cc:1060] running on GCE node
I20260128 20:51:35.453999 13253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:35.455152 13253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:35.456408 13253 hybrid_clock.cc:648] HybridClock initialized: now 1769633495456386 us; error 35 us; skew 500 ppm
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:35.459481 13253 init.cc:377] Logged in from keytab as oryx/127.4.183.130@KRBTEST.COM (short username oryx)
I20260128 20:51:35.460757 13253 webserver.cc:492] Webserver started at http://127.4.183.130:42611/ using document root <none> and password file <none>
I20260128 20:51:35.461432 13253 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:35.461490 13253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:35.461681 13253 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:35.463598 13253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "e135aeabf5894f10950acf03519e9730"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "d720737c76e3eb595168d2d54c0ede39"
server_key_iv: "41557448284412ab48b09d9ce6183ced"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.464131 13253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "e135aeabf5894f10950acf03519e9730"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "d720737c76e3eb595168d2d54c0ede39"
server_key_iv: "41557448284412ab48b09d9ce6183ced"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.468010 13253 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:51:35.470860 13268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:35.472100 13253 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:35.472230 13253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "e135aeabf5894f10950acf03519e9730"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "d720737c76e3eb595168d2d54c0ede39"
server_key_iv: "41557448284412ab48b09d9ce6183ced"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.472390 13253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:35.488929 13253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:35.492427 13253 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:35.492693 13253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:35.494351 13253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:35.494413 13253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:35.494457 13253 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:35.494524 13253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:35.507587 13253 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:40477
I20260128 20:51:35.507606 13383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:40477 every 8 connection(s)
I20260128 20:51:35.508724 13253 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:35.510478 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.495355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:39041 (local address 127.4.183.190:34393)
0128 20:51:35.495507 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.495510 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.496489 (+ 979us) server_negotiation.cc:408] Connection header received
0128 20:51:35.497567 (+ 1078us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.497571 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.497647 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.497757 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.499884 (+ 2127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.500743 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.501866 (+ 1123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.502089 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.506389 (+ 4300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.506422 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.506425 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.506460 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.508151 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.508734 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.508738 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.508742 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.508825 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.509173 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.509178 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.509181 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.509419 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.509524 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.510208 (+ 684us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.510347 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 20:51:35.511183 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.495779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.4.183.130:39041)
0128 20:51:35.496331 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.496372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.497329 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.498043 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.498059 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.498747 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:35.499631 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.499666 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.500925 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.500932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.501706 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.501720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.502037 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.502874 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:35.502925 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:35.506158 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:35.508314 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.508321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.508335 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.508607 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.508936 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.508945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.508947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.509043 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.509585 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:35.509591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:35.509898 (+ 307us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.510130 (+ 232us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.510384 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":162,"threads_started":1}
I20260128 20:51:35.512596 13384 heartbeater.cc:344] Connected to a master server at 127.4.183.190:34393
I20260128 20:51:35.512799 13384 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:35.513562 13384 heartbeater.cc:507] Master 127.4.183.190:34393 requested a full tablet report, sending...
I20260128 20:51:35.514922 13037 ts_manager.cc:194] Registered new tserver with Master: e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477)
I20260128 20:51:35.515821 13037 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:39041
I20260128 20:51:35.517791 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13253
I20260128 20:51:35.517927 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:35.518185 4830 external_mini_cluster.cc:1468] Setting key fd0a59565cc9c1737b42f8ff6624f413
I20260128 20:51:35.522773 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.515254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.4.183.130:55565)
0128 20:51:35.515634 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.515651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.515763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.516070 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.516073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.516306 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:35.516625 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.516656 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.517449 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.517454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.517924 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.517939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.518128 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.518811 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:35.518826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:35.519152 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:35.520979 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.520984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.520987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.521307 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.521610 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.521616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.521621 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.521696 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.522258 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:35.522262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:35.522361 (+ 99us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.522465 (+ 104us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.522606 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260128 20:51:35.522961 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.515313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:55565 (local address 127.4.183.190:34393)
0128 20:51:35.515489 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.515494 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.515729 (+ 235us) server_negotiation.cc:408] Connection header received
0128 20:51:35.515868 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.515871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.515945 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.516047 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.516759 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.517324 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.518106 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.518320 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.519288 (+ 968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.519307 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.519315 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.519340 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.520819 (+ 1479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.521444 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.521448 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.521449 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.521502 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.521811 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.521814 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.521815 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.522124 (+ 309us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.522237 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.522551 (+ 314us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.522708 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for oryx/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab.
I20260128 20:51:35.581741 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:34393
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:35.705966 13395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:35.706337 13395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:35.706444 13395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:35.710098 13395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:35.710235 13395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:35.710337 13395 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 20:51:35.710441 13395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:35.715480 13395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/krb5kdc/oryx.keytab
--principal=oryx/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.183.190:34393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:35.716811 13395 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:35.717866 13395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:35.725556 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:35.725625 13401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:35.725577 13403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:35.725960 13395 server_base.cc:1060] running on GCE node
I20260128 20:51:35.726516 13395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:35.727173 13395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:35.728374 13395 hybrid_clock.cc:648] HybridClock initialized: now 1769633495728356 us; error 35 us; skew 500 ppm
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:35.731739 13395 init.cc:377] Logged in from keytab as oryx/127.4.183.131@KRBTEST.COM (short username oryx)
I20260128 20:51:35.733073 13395 webserver.cc:492] Webserver started at http://127.4.183.131:34329/ using document root <none> and password file <none>
I20260128 20:51:35.733790 13395 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:35.733866 13395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:35.734052 13395 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:35.736028 13395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "824f0a5f93bfba591252d550f7ac1f5d"
server_key_iv: "45fde87aa910d16f4798fd6169ae3034"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.736537 13395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "824f0a5f93bfba591252d550f7ac1f5d"
server_key_iv: "45fde87aa910d16f4798fd6169ae3034"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.740437 13395 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260128 20:51:35.743470 13410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:35.744894 13395 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260128 20:51:35.745045 13395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2"
format_stamp: "Formatted at 2026-01-28 20:51:35 on dist-test-slave-hss7"
server_key: "824f0a5f93bfba591252d550f7ac1f5d"
server_key_iv: "45fde87aa910d16f4798fd6169ae3034"
server_key_version: "encryptionkey@0"
I20260128 20:51:35.745255 13395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:35.767139 13395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:35.770654 13395 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:35.770939 13395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:35.773435 13395 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:35.773559 13395 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:35.773638 13395 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:35.773674 13395 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:35.788271 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.773917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:37673 (local address 127.4.183.190:34393)
0128 20:51:35.774110 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.774115 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.774833 (+ 718us) server_negotiation.cc:408] Connection header received
0128 20:51:35.775794 (+ 961us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.775798 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.775881 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.776000 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.778707 (+ 2707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.779239 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.780025 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.780210 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.783592 (+ 3382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.783622 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.783627 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.783666 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.785602 (+ 1936us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.786403 (+ 801us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.786410 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.786415 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.786493 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.786875 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.786879 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.786881 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.787176 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.787305 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.787936 (+ 631us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.788091 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260128 20:51:35.788368 13395 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:41403
I20260128 20:51:35.788385 13525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:41403 every 8 connection(s)
I20260128 20:51:35.789094 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.774135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.4.183.131:37673)
0128 20:51:35.774636 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.774684 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.775556 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.776329 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.776358 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.777296 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:35.778462 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.778483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.779409 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.779415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.779852 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.779861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.780169 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.781005 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:35.781065 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:35.783370 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:35.785793 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.785800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.785812 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.786284 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.786610 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.786614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.786617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.786754 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.787317 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:35.787327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:35.787620 (+ 293us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.787882 (+ 262us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.788199 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":142,"threads_started":1}
I20260128 20:51:35.789676 13395 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:35.790633 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13395
I20260128 20:51:35.790760 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:35.791091 4830 external_mini_cluster.cc:1468] Setting key a8652075b99590733878ff7add863577
I20260128 20:51:35.791493 13526 heartbeater.cc:344] Connected to a master server at 127.4.183.190:34393
I20260128 20:51:35.792450 13526 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:35.793223 13526 heartbeater.cc:507] Master 127.4.183.190:34393 requested a full tablet report, sending...
I20260128 20:51:35.794476 13037 ts_manager.cc:194] Registered new tserver with Master: 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:35.795286 13037 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:37673
I20260128 20:51:35.802834 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.795903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.4.183.131:52999)
0128 20:51:35.796231 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.796250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.796348 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.796647 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.796651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.796871 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:35.797090 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.797097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.798141 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.798145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.798530 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.798539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.798650 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.799225 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:35.799238 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:35.799603 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:35.801555 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.801558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.801560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.801760 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.802012 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:35.802014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:35.802016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:35.802061 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:35.802375 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:35.802377 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:35.802459 (+ 82us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.802542 (+ 83us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.802672 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":147,"threads_started":1}
I20260128 20:51:35.803103 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.795635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:52999 (local address 127.4.183.190:34393)
0128 20:51:35.795789 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.795794 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.796338 (+ 544us) server_negotiation.cc:408] Connection header received
0128 20:51:35.796443 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.796446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.796502 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.796593 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.797278 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.797994 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.798665 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.798821 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.799751 (+ 930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.799771 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.799773 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.799799 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.801400 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.801873 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.801876 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.801877 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.801926 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.802136 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.802138 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.802139 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.802300 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.802374 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.802720 (+ 346us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.802831 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260128 20:51:35.806911 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 20:51:35.833598 13086 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:35.826530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.4.183.190:34393)
0128 20:51:35.826708 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.826711 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.826730 (+ 19us) server_negotiation.cc:408] Connection header received
0128 20:51:35.826795 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.826799 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.826861 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.826980 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.828164 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.828725 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.829594 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.829777 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.833412 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47906: BlockingRecv error: recv got EOF from 127.0.0.1:47906 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jan 28 20:51:35 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:35.844024 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.835051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.4.183.190:34393)
0128 20:51:35.835184 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.835187 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.835202 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:35.835250 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.835253 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.835300 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.835379 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:35.836236 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.836781 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.837733 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.837897 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.840575 (+ 2678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:35.840599 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:35.840602 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:35.840668 (+ 66us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:35.842302 (+ 1634us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.842884 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.842887 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.842888 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.842936 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:35.843162 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:35.843164 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:35.843165 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:35.843334 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:35.843442 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.843776 (+ 334us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.843890 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 20:51:35.847589 13037 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:35.850915 13037 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:35.867761 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.863234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:39527 (local address 127.0.0.1:44600)
0128 20:51:35.863651 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.863698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.863895 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.864537 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.864544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.864573 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:35.864928 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.864937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.866357 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.866364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.867205 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.867218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.867326 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.867417 (+ 91us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.867526 (+ 109us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.867625 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":116,"threads_started":1}
I20260128 20:51:35.868623 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.863758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:40477 (local address 127.0.0.1:33716)
0128 20:51:35.864523 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.864537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.864602 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.865389 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.865392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.865416 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:35.865726 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.865735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.867186 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.867190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.868201 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.868213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.868335 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.868357 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.868409 (+ 52us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.868477 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":108,"threads_started":1}
I20260128 20:51:35.868981 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.864051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:41403 (local address 127.0.0.1:40270)
0128 20:51:35.864780 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.864791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:35.864918 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:35.865541 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:35.865545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:35.865563 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:35.865801 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.865808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.867471 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.867475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:35.868575 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:35.868586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.868712 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.868738 (+ 26us) client_negotiation.cc:769] Sending connection context
0128 20:51:35.868793 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 20:51:35.868850 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":94,"threads_started":1}
I20260128 20:51:35.869216 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.863384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.4.183.129:39527)
0128 20:51:35.864027 (+ 643us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.864037 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.864061 (+ 24us) server_negotiation.cc:408] Connection header received
0128 20:51:35.864145 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.864150 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.864352 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.865086 (+ 734us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:35.865263 (+ 177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.866229 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.868053 (+ 1824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.868644 (+ 591us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.868801 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.868893 (+ 92us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.868998 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":76,"threads_started":1}
I20260128 20:51:35.870299 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.864277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.4.183.131:41403)
0128 20:51:35.864661 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.864669 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.864889 (+ 220us) server_negotiation.cc:408] Connection header received
0128 20:51:35.865073 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.865119 (+ 46us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.865351 (+ 232us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.865499 (+ 148us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:35.865938 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.867269 (+ 1331us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.869310 (+ 2041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.869838 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.869975 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.870049 (+ 74us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.870152 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
I20260128 20:51:35.871160 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.864437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.4.183.130:40477)
0128 20:51:35.864866 (+ 429us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.864873 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:35.864894 (+ 21us) server_negotiation.cc:408] Connection header received
0128 20:51:35.864975 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:35.864982 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:35.865210 (+ 228us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:35.865361 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:35.865870 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.867034 (+ 1164us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:35.868393 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:35.869226 (+ 833us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:35.870203 (+ 977us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:35.870272 (+ 69us) server_negotiation.cc:300] Negotiation successful
0128 20:51:35.870376 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":114,"threads_started":1}
I20260128 20:51:35.872751 13172 tablet_service.cc:1505] Processing CreateTablet for tablet 57824fe88eeb438794ee78877f23462f (DEFAULT_TABLE table=test-table [id=68cc466c19034f688cba4a618443e04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:35.872865 13460 tablet_service.cc:1505] Processing CreateTablet for tablet 57824fe88eeb438794ee78877f23462f (DEFAULT_TABLE table=test-table [id=68cc466c19034f688cba4a618443e04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:35.873890 13460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57824fe88eeb438794ee78877f23462f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:35.873903 13172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57824fe88eeb438794ee78877f23462f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:35.874007 13318 tablet_service.cc:1505] Processing CreateTablet for tablet 57824fe88eeb438794ee78877f23462f (DEFAULT_TABLE table=test-table [id=68cc466c19034f688cba4a618443e04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:35.875162 13318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57824fe88eeb438794ee78877f23462f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:35.880574 13554 tablet_bootstrap.cc:492] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Bootstrap starting.
I20260128 20:51:35.882237 13555 tablet_bootstrap.cc:492] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Bootstrap starting.
I20260128 20:51:35.884102 13554 tablet_bootstrap.cc:654] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:35.884498 13555 tablet_bootstrap.cc:654] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:35.885414 13555 log.cc:826] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:35.885416 13554 log.cc:826] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:35.885794 13556 tablet_bootstrap.cc:492] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Bootstrap starting.
I20260128 20:51:35.887715 13554 tablet_bootstrap.cc:492] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: No bootstrap required, opened a new log
I20260128 20:51:35.887742 13555 tablet_bootstrap.cc:492] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: No bootstrap required, opened a new log
I20260128 20:51:35.887941 13554 ts_tablet_manager.cc:1403] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260128 20:51:35.887940 13555 ts_tablet_manager.cc:1403] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260128 20:51:35.888247 13556 tablet_bootstrap.cc:654] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:35.889617 13556 log.cc:826] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:35.891243 13555 raft_consensus.cc:359] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.891273 13554 raft_consensus.cc:359] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.891465 13555 raft_consensus.cc:385] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:35.891495 13554 raft_consensus.cc:385] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:35.891503 13555 raft_consensus.cc:740] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Initialized, Role: FOLLOWER
I20260128 20:51:35.891548 13554 raft_consensus.cc:740] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Initialized, Role: FOLLOWER
I20260128 20:51:35.892107 13555 consensus_queue.cc:260] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.893214 13555 ts_tablet_manager.cc:1434] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260128 20:51:35.893637 13242 heartbeater.cc:499] Master 127.4.183.190:34393 was elected leader, sending a full tablet report...
I20260128 20:51:35.893786 13554 consensus_queue.cc:260] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.894111 13556 tablet_bootstrap.cc:492] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: No bootstrap required, opened a new log
I20260128 20:51:35.894354 13556 ts_tablet_manager.cc:1403] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260128 20:51:35.894997 13554 ts_tablet_manager.cc:1434] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260128 20:51:35.898027 13556 raft_consensus.cc:359] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.898378 13556 raft_consensus.cc:385] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:35.898450 13556 raft_consensus.cc:740] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e135aeabf5894f10950acf03519e9730, State: Initialized, Role: FOLLOWER
I20260128 20:51:35.898958 13526 heartbeater.cc:499] Master 127.4.183.190:34393 was elected leader, sending a full tablet report...
I20260128 20:51:35.900627 13556 consensus_queue.cc:260] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.901660 13384 heartbeater.cc:499] Master 127.4.183.190:34393 was elected leader, sending a full tablet report...
I20260128 20:51:35.903199 13556 ts_tablet_manager.cc:1434] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260128 20:51:35.992203 13243 tablet.cc:2378] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:35.997762 13560 raft_consensus.cc:493] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:35.997967 13560 raft_consensus.cc:515] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:35.999078 13560 leader_election.cc:290] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477), 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:36.002796 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.999444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:41403 (local address 127.4.183.129:51881)
0128 20:51:35.999911 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:35.999927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.000096 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.000391 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.000394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.000413 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:36.000714 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.000723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.001632 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.001635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.002273 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.002282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.002436 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.002460 (+ 24us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.002515 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.002599 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":177,"threads_started":1}
I20260128 20:51:36.002799 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.999444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:40477 (local address 127.4.183.129:43891)
0128 20:51:35.999995 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.000008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.000146 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.000445 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.000449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.000466 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:36.000746 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.000754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.001628 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.001632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.002273 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.002282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.002434 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.002460 (+ 26us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.002515 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.002591 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":206,"thread_start_us":163,"threads_started":1}
I20260128 20:51:36.003283 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.999561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:43891 (local address 127.4.183.130:40477)
0128 20:51:35.999716 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.999724 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.000106 (+ 382us) server_negotiation.cc:408] Connection header received
0128 20:51:36.000274 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.000277 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.000335 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.000432 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:36.000884 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.001470 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.002411 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.002955 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.003002 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.003059 (+ 57us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.003138 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 20:51:36.003288 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:35.999577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:51881 (local address 127.4.183.131:41403)
0128 20:51:35.999732 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:35.999737 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.000028 (+ 291us) server_negotiation.cc:408] Connection header received
0128 20:51:36.000216 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.000220 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.000278 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.000376 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:36.000856 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.001470 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.002410 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.002940 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.002993 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.003054 (+ 61us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.003132 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260128 20:51:36.004024 13480 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57824fe88eeb438794ee78877f23462f" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" is_pre_election: true
I20260128 20:51:36.004024 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57824fe88eeb438794ee78877f23462f" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730" is_pre_election: true
I20260128 20:51:36.004341 13480 raft_consensus.cc:2468] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 0.
I20260128 20:51:36.004341 13338 raft_consensus.cc:2468] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 0.
I20260128 20:51:36.004992 13129 leader_election.cc:304] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6e83fc466c4e169f49e61dda87d042, e135aeabf5894f10950acf03519e9730; no voters:
I20260128 20:51:36.005359 13560 raft_consensus.cc:2804] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:36.005465 13560 raft_consensus.cc:493] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:36.005524 13560 raft_consensus.cc:3060] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.006574 13560 raft_consensus.cc:515] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.007046 13560 leader_election.cc:290] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 election: Requested vote from peers e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477), 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:36.007861 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57824fe88eeb438794ee78877f23462f" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730"
I20260128 20:51:36.007861 13480 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57824fe88eeb438794ee78877f23462f" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2"
I20260128 20:51:36.008028 13338 raft_consensus.cc:3060] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.008028 13480 raft_consensus.cc:3060] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.009486 13480 raft_consensus.cc:2468] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 1.
I20260128 20:51:36.009486 13338 raft_consensus.cc:2468] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 1.
W20260128 20:51:36.009987 13385 tablet.cc:2378] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:36.009994 13128 leader_election.cc:304] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, 7f6e83fc466c4e169f49e61dda87d042; no voters:
I20260128 20:51:36.010303 13560 raft_consensus.cc:2804] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:36.010604 13560 raft_consensus.cc:697] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 1 LEADER]: Becoming Leader. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Running, Role: LEADER
I20260128 20:51:36.011019 13560 consensus_queue.cc:237] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.014724 13037 catalog_manager.cc:5654] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129). New cstate: current_term: 1 leader_uuid: "7f6e83fc466c4e169f49e61dda87d042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } health_report { overall_health: UNKNOWN } } }
I20260128 20:51:36.034274 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.030773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.4.183.129:39527)
0128 20:51:36.030927 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.030930 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.030944 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:36.031033 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.031036 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.031095 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.031185 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.031584 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.032119 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.032835 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.033008 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.033083 (+ 75us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.033750 (+ 667us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.033890 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.034022 (+ 132us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.034086 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260128 20:51:36.040714 13527 tablet.cc:2378] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:51:36.041762 13480 raft_consensus.cc:1275] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Refusing update from remote peer 7f6e83fc466c4e169f49e61dda87d042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.041762 13338 raft_consensus.cc:1275] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Refusing update from remote peer 7f6e83fc466c4e169f49e61dda87d042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.042678 13560 consensus_queue.cc:1048] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.042994 13565 consensus_queue.cc:1048] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.063628 13037 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:36.063890 13037 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:36.067675 13037 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Sending DeleteTablet for 3 replicas of tablet 57824fe88eeb438794ee78877f23462f
I20260128 20:51:36.068783 13172 tablet_service.cc:1552] Processing DeleteTablet for tablet 57824fe88eeb438794ee78877f23462f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:44600
I20260128 20:51:36.068966 13460 tablet_service.cc:1552] Processing DeleteTablet for tablet 57824fe88eeb438794ee78877f23462f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:40270
I20260128 20:51:36.069299 13318 tablet_service.cc:1552] Processing DeleteTablet for tablet 57824fe88eeb438794ee78877f23462f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:33716
I20260128 20:51:36.069777 13578 tablet_replica.cc:333] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: stopping tablet replica
I20260128 20:51:36.069806 13579 tablet_replica.cc:333] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: stopping tablet replica
I20260128 20:51:36.069887 13037 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 20:51:36.070101 13578 raft_consensus.cc:2243] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:36.070101 13579 raft_consensus.cc:2243] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260128 20:51:36.070464 13037 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:36.070475 13578 raft_consensus.cc:2272] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.070475 13579 raft_consensus.cc:2272] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.072163 13579 ts_tablet_manager.cc:1916] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.073441 13580 tablet_replica.cc:333] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: stopping tablet replica
I20260128 20:51:36.073485 13578 ts_tablet_manager.cc:1916] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.073920 13580 raft_consensus.cc:2243] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:36.074517 13580 raft_consensus.cc:2272] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.076192 13580 ts_tablet_manager.cc:1916] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.077867 13460 tablet_service.cc:1505] Processing CreateTablet for tablet 00aeeda83f5242d68d0a8bd66f0eae62 (DEFAULT_TABLE table=test-table [id=c6f5b7a4a3264dba8b096d15f3381513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.078243 13460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00aeeda83f5242d68d0a8bd66f0eae62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.078876 13579 ts_tablet_manager.cc:1929] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:36.079001 13579 log.cc:1199] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/57824fe88eeb438794ee78877f23462f
I20260128 20:51:36.079329 13579 ts_tablet_manager.cc:1950] T 57824fe88eeb438794ee78877f23462f P e135aeabf5894f10950acf03519e9730: Deleting consensus metadata
I20260128 20:51:36.079726 13580 ts_tablet_manager.cc:1929] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:36.079916 13580 log.cc:1199] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/57824fe88eeb438794ee78877f23462f
I20260128 20:51:36.080325 13580 ts_tablet_manager.cc:1950] T 57824fe88eeb438794ee78877f23462f P 7f6e83fc466c4e169f49e61dda87d042: Deleting consensus metadata
I20260128 20:51:36.081174 13172 tablet_service.cc:1505] Processing CreateTablet for tablet 00aeeda83f5242d68d0a8bd66f0eae62 (DEFAULT_TABLE table=test-table [id=c6f5b7a4a3264dba8b096d15f3381513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.081470 13172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00aeeda83f5242d68d0a8bd66f0eae62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.081815 13578 ts_tablet_manager.cc:1929] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:36.081931 13578 log.cc:1199] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/57824fe88eeb438794ee78877f23462f
I20260128 20:51:36.082238 13578 ts_tablet_manager.cc:1950] T 57824fe88eeb438794ee78877f23462f P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting consensus metadata
I20260128 20:51:36.083491 13555 tablet_bootstrap.cc:492] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Bootstrap starting.
I20260128 20:51:36.083957 13022 catalog_manager.cc:4985] TS 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403): tablet 57824fe88eeb438794ee78877f23462f (table test-table [id=68cc466c19034f688cba4a618443e04d]) successfully deleted
I20260128 20:51:36.084185 13022 catalog_manager.cc:4985] TS 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527): tablet 57824fe88eeb438794ee78877f23462f (table test-table [id=68cc466c19034f688cba4a618443e04d]) successfully deleted
I20260128 20:51:36.084542 13023 catalog_manager.cc:4985] TS e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477): tablet 57824fe88eeb438794ee78877f23462f (table test-table [id=68cc466c19034f688cba4a618443e04d]) successfully deleted
I20260128 20:51:36.084810 13555 tablet_bootstrap.cc:654] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.084856 13554 tablet_bootstrap.cc:492] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Bootstrap starting.
I20260128 20:51:36.085505 13318 tablet_service.cc:1505] Processing CreateTablet for tablet 00aeeda83f5242d68d0a8bd66f0eae62 (DEFAULT_TABLE table=test-table [id=c6f5b7a4a3264dba8b096d15f3381513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.085810 13318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00aeeda83f5242d68d0a8bd66f0eae62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.085956 13554 tablet_bootstrap.cc:654] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.087399 13554 tablet_bootstrap.cc:492] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: No bootstrap required, opened a new log
I20260128 20:51:36.087496 13554 ts_tablet_manager.cc:1403] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:36.088088 13554 raft_consensus.cc:359] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.088217 13554 raft_consensus.cc:385] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.088253 13554 raft_consensus.cc:740] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.088354 13556 tablet_bootstrap.cc:492] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Bootstrap starting.
I20260128 20:51:36.088429 13554 consensus_queue.cc:260] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.089323 13554 ts_tablet_manager.cc:1434] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260128 20:51:36.089543 13556 tablet_bootstrap.cc:654] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.090994 13555 tablet_bootstrap.cc:492] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: No bootstrap required, opened a new log
I20260128 20:51:36.091140 13555 ts_tablet_manager.cc:1403] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260128 20:51:36.091710 13555 raft_consensus.cc:359] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.091854 13555 raft_consensus.cc:385] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.091903 13555 raft_consensus.cc:740] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.092046 13555 consensus_queue.cc:260] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.092307 13555 ts_tablet_manager.cc:1434] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260128 20:51:36.092514 13556 tablet_bootstrap.cc:492] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: No bootstrap required, opened a new log
I20260128 20:51:36.092605 13556 ts_tablet_manager.cc:1403] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260128 20:51:36.093106 13556 raft_consensus.cc:359] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.093302 13556 raft_consensus.cc:385] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.093336 13556 raft_consensus.cc:740] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e135aeabf5894f10950acf03519e9730, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.093494 13556 consensus_queue.cc:260] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.094198 13556 ts_tablet_manager.cc:1434] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260128 20:51:36.167029 13573 raft_consensus.cc:493] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:36.167202 13573 raft_consensus.cc:515] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.167824 13573 leader_election.cc:290] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477), 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:36.168341 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00aeeda83f5242d68d0a8bd66f0eae62" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730" is_pre_election: true
I20260128 20:51:36.168368 13480 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00aeeda83f5242d68d0a8bd66f0eae62" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" is_pre_election: true
I20260128 20:51:36.168529 13338 raft_consensus.cc:2468] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 0.
I20260128 20:51:36.168535 13480 raft_consensus.cc:2468] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 0.
I20260128 20:51:36.168900 13128 leader_election.cc:304] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, 7f6e83fc466c4e169f49e61dda87d042; no voters:
I20260128 20:51:36.169163 13573 raft_consensus.cc:2804] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:36.169255 13573 raft_consensus.cc:493] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:36.169315 13573 raft_consensus.cc:3060] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.170387 13573 raft_consensus.cc:515] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.170878 13573 leader_election.cc:290] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 election: Requested vote from peers e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477), 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:36.171362 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00aeeda83f5242d68d0a8bd66f0eae62" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730"
I20260128 20:51:36.171362 13480 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00aeeda83f5242d68d0a8bd66f0eae62" candidate_uuid: "7f6e83fc466c4e169f49e61dda87d042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2"
I20260128 20:51:36.171514 13338 raft_consensus.cc:3060] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.171514 13480 raft_consensus.cc:3060] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.172761 13338 raft_consensus.cc:2468] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 1.
I20260128 20:51:36.172761 13480 raft_consensus.cc:2468] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6e83fc466c4e169f49e61dda87d042 in term 1.
I20260128 20:51:36.173264 13128 leader_election.cc:304] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, 7f6e83fc466c4e169f49e61dda87d042; no voters:
I20260128 20:51:36.173538 13573 raft_consensus.cc:2804] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:36.173669 13573 raft_consensus.cc:697] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 LEADER]: Becoming Leader. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Running, Role: LEADER
I20260128 20:51:36.173856 13573 consensus_queue.cc:237] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.176070 13035 catalog_manager.cc:5654] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129). New cstate: current_term: 1 leader_uuid: "7f6e83fc466c4e169f49e61dda87d042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } health_report { overall_health: UNKNOWN } } }
Jan 28 20:51:36 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633494, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.190@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:36.229696 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.219180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:34393 (local address 127.0.0.1:47926)
0128 20:51:36.219533 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.219551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.219695 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.219998 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.220001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.220581 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:36.220872 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.220883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.221732 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.221737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.222345 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.222355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.222497 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.223202 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:36.223221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:36.225170 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:36.227643 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:36.227659 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:36.227666 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:36.228040 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:36.228354 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:36.228358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:36.228360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:36.228424 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:36.228912 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:36.228921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:36.229051 (+ 130us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.229230 (+ 179us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.229434 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
I20260128 20:51:36.229684 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.219277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.4.183.190:34393)
0128 20:51:36.219441 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.219446 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.219708 (+ 262us) server_negotiation.cc:408] Connection header received
0128 20:51:36.219811 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.219813 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.219859 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.219936 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:36.221049 (+ 1113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.221571 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.222513 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.222663 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.225424 (+ 2761us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:36.225622 (+ 198us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:36.225633 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:36.225671 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:36.227442 (+ 1771us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:36.228179 (+ 737us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:36.228183 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:36.228184 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:36.228233 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:36.228555 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:36.228567 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:36.228569 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:36.228758 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:36.228851 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.229317 (+ 466us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.229518 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":74}
I20260128 20:51:36.236253 13035 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:47926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260128 20:51:36.237310 13035 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:36.245023 13172 tablet_service.cc:1505] Processing CreateTablet for tablet 91e6c08d144f47d0ac7dc9783166d26a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cadf76a84f584b27af4bf41e3e579779]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:51:36.245332 13318 tablet_service.cc:1505] Processing CreateTablet for tablet 91e6c08d144f47d0ac7dc9783166d26a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cadf76a84f584b27af4bf41e3e579779]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:51:36.245836 13318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91e6c08d144f47d0ac7dc9783166d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.245939 13172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91e6c08d144f47d0ac7dc9783166d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.247514 13460 tablet_service.cc:1505] Processing CreateTablet for tablet 91e6c08d144f47d0ac7dc9783166d26a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cadf76a84f584b27af4bf41e3e579779]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 20:51:36.248114 13460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91e6c08d144f47d0ac7dc9783166d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.249049 13556 tablet_bootstrap.cc:492] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: Bootstrap starting.
I20260128 20:51:36.249061 13555 tablet_bootstrap.cc:492] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: Bootstrap starting.
I20260128 20:51:36.250336 13555 tablet_bootstrap.cc:654] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.250413 13556 tablet_bootstrap.cc:654] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.252094 13554 tablet_bootstrap.cc:492] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: Bootstrap starting.
I20260128 20:51:36.253306 13554 tablet_bootstrap.cc:654] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.255070 13555 tablet_bootstrap.cc:492] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: No bootstrap required, opened a new log
I20260128 20:51:36.255203 13555 ts_tablet_manager.cc:1403] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260128 20:51:36.255271 13554 tablet_bootstrap.cc:492] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: No bootstrap required, opened a new log
I20260128 20:51:36.255390 13554 ts_tablet_manager.cc:1403] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260128 20:51:36.255771 13555 raft_consensus.cc:359] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.255909 13555 raft_consensus.cc:385] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.255947 13555 raft_consensus.cc:740] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.256028 13554 raft_consensus.cc:359] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.256104 13555 consensus_queue.cc:260] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.256167 13554 raft_consensus.cc:385] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.256207 13554 raft_consensus.cc:740] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.256356 13573 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.256491 13573 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.257364 13555 ts_tablet_manager.cc:1434] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260128 20:51:36.258056 13556 tablet_bootstrap.cc:492] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: No bootstrap required, opened a new log
I20260128 20:51:36.258164 13556 ts_tablet_manager.cc:1403] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260128 20:51:36.258765 13556 raft_consensus.cc:359] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.258924 13556 raft_consensus.cc:385] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.258993 13556 raft_consensus.cc:740] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e135aeabf5894f10950acf03519e9730, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.259145 13556 consensus_queue.cc:260] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.259389 13554 consensus_queue.cc:260] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.259740 13554 ts_tablet_manager.cc:1434] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260128 20:51:36.260023 13562 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.260169 13562 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.261380 13556 ts_tablet_manager.cc:1434] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:36.261412 13561 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.261560 13561 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.470006 13561 raft_consensus.cc:493] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:36.470219 13561 raft_consensus.cc:515] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.471405 13561 leader_election.cc:290] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527), e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477)
I20260128 20:51:36.474858 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.471728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:40477 (local address 127.4.183.131:39933)
0128 20:51:36.472072 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.472087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.472201 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.472514 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.472517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.472560 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:36.472805 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.472811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.473686 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.473695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.474324 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.474332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.474483 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.474510 (+ 27us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.474566 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.474641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20260128 20:51:36.475219 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.471848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:39933 (local address 127.4.183.130:40477)
0128 20:51:36.472003 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.472009 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.472191 (+ 182us) server_negotiation.cc:408] Connection header received
0128 20:51:36.472337 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.472340 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.472387 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.472483 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:36.472956 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.473557 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.474458 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.474961 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.475009 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.475078 (+ 69us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.475131 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:51:36.475260 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.471866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:50749 (local address 127.4.183.129:39527)
0128 20:51:36.472063 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.472068 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.472090 (+ 22us) server_negotiation.cc:408] Connection header received
0128 20:51:36.472314 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.472318 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.472371 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.472436 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:36.472897 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.473555 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.474462 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.474977 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.475024 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.475084 (+ 60us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.475145 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260128 20:51:36.474858 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.471619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:39527 (local address 127.4.183.131:50749)
0128 20:51:36.472002 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.472021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.472166 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.472481 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.472484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.472503 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:36.472763 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.472771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.473681 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.473686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.474323 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.474332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.474485 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.474511 (+ 26us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.474567 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.474638 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"spinlock_wait_cycles":24704,"thread_start_us":89,"threads_started":1}
I20260128 20:51:36.476197 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91e6c08d144f47d0ac7dc9783166d26a" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730" is_pre_election: true
I20260128 20:51:36.476293 13196 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91e6c08d144f47d0ac7dc9783166d26a" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e83fc466c4e169f49e61dda87d042" is_pre_election: true
I20260128 20:51:36.476594 13338 raft_consensus.cc:2468] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 0.
I20260128 20:51:36.476634 13196 raft_consensus.cc:2468] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 0.
I20260128 20:51:36.477291 13413 leader_election.cc:304] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, e135aeabf5894f10950acf03519e9730; no voters:
I20260128 20:51:36.477825 13561 raft_consensus.cc:2804] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:36.477911 13561 raft_consensus.cc:493] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:36.477944 13561 raft_consensus.cc:3060] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.479393 13561 raft_consensus.cc:515] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.479967 13561 leader_election.cc:290] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527), e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477)
I20260128 20:51:36.480551 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91e6c08d144f47d0ac7dc9783166d26a" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730"
I20260128 20:51:36.480841 13338 raft_consensus.cc:3060] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.481488 13196 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91e6c08d144f47d0ac7dc9783166d26a" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e83fc466c4e169f49e61dda87d042"
I20260128 20:51:36.481753 13196 raft_consensus.cc:3060] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.482396 13338 raft_consensus.cc:2468] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 1.
I20260128 20:51:36.482860 13196 raft_consensus.cc:2468] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 1.
I20260128 20:51:36.482944 13413 leader_election.cc:304] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, e135aeabf5894f10950acf03519e9730; no voters:
I20260128 20:51:36.483246 13561 raft_consensus.cc:2804] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:36.483456 13561 raft_consensus.cc:697] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 LEADER]: Becoming Leader. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Running, Role: LEADER
I20260128 20:51:36.483758 13561 consensus_queue.cc:237] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.485118 13596 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: TxnStatusTablet state changed. Reason: New leader 3a0aaa6f423c4e69aab2f9b662cf78e2. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.485287 13596 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:51:36.485574 13598 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260128 20:51:36.485718 13598 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260128 20:51:36.486693 13035 catalog_manager.cc:5654] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131). New cstate: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } health_report { overall_health: HEALTHY } } }
I20260128 20:51:36.507004 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.502296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:41403 (local address 127.0.0.1:40286)
0128 20:51:36.502458 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.502475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.502606 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.503018 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.503022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.503031 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:51:36.503330 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.503338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.504086 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.504091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.504764 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.504775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.504907 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.504967 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.506539 (+ 1572us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.506570 (+ 31us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.506690 (+ 120us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.506805 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260128 20:51:36.507048 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.502401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.4.183.131:41403)
0128 20:51:36.502534 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.502539 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.502558 (+ 19us) server_negotiation.cc:408] Connection header received
0128 20:51:36.502812 (+ 254us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.502815 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.502877 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.502982 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.503434 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.503971 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.504970 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.505306 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.505411 (+ 105us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.506366 (+ 955us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.506529 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.506762 (+ 233us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.506916 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 20:51:36.510644 13338 raft_consensus.cc:1275] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Refusing update from remote peer 3a0aaa6f423c4e69aab2f9b662cf78e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.511003 13196 raft_consensus.cc:1275] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Refusing update from remote peer 3a0aaa6f423c4e69aab2f9b662cf78e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.511776 13596 consensus_queue.cc:1048] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.512775 13596 consensus_queue.cc:1048] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.515856 13562 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: TxnStatusTablet state changed. Reason: New leader 3a0aaa6f423c4e69aab2f9b662cf78e2. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.516007 13562 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.516057 13573 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: TxnStatusTablet state changed. Reason: New leader 3a0aaa6f423c4e69aab2f9b662cf78e2. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.516171 13573 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.518649 13561 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.518805 13561 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:51:36.521507 13570 mvcc.cc:204] Tried to move back new op lower bound from 7248418801701879808 to 7248418801600450560. Current Snapshot: MvccSnapshot[applied={T|T < 7248418801701879808 or (T in {7248418801701879808})}]
I20260128 20:51:36.522158 13562 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.522289 13562 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P e135aeabf5894f10950acf03519e9730: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.527305 13561 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.527457 13561 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 3a0aaa6f423c4e69aab2f9b662cf78e2: This TxnStatusTablet replica's current role is: LEADER
I20260128 20:51:36.529006 13573 tablet_replica.cc:442] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } } }
I20260128 20:51:36.529177 13573 tablet_replica.cc:445] T 91e6c08d144f47d0ac7dc9783166d26a P 7f6e83fc466c4e169f49e61dda87d042: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 20:51:36.529480 13568 mvcc.cc:204] Tried to move back new op lower bound from 7248418801701879808 to 7248418801600450560. Current Snapshot: MvccSnapshot[applied={T|T < 7248418801701879808 or (T in {7248418801701879808})}]
I20260128 20:51:36.538620 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.534773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:59909 (local address 127.4.183.131:41403)
0128 20:51:36.534949 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.534954 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.535146 (+ 192us) server_negotiation.cc:408] Connection header received
0128 20:51:36.535311 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.535314 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.535369 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.535444 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.535971 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.536482 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.537643 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.537880 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.537946 (+ 66us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.538066 (+ 120us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.538184 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.538420 (+ 236us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.538501 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 20:51:36.538648 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.534680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:41403 (local address 127.4.183.129:59909)
0128 20:51:36.535044 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.535061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.535200 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.535478 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.535482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.535491 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:51:36.535813 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.535823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.536616 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.536621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.537332 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.537346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.537510 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.537582 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.538236 (+ 654us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.538244 (+ 8us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.538345 (+ 101us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.538453 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":144,"threads_started":1}
I20260128 20:51:36.548271 13338 raft_consensus.cc:1275] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Refusing update from remote peer 7f6e83fc466c4e169f49e61dda87d042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.548274 13480 raft_consensus.cc:1275] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Refusing update from remote peer 7f6e83fc466c4e169f49e61dda87d042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.548843 13565 consensus_queue.cc:1048] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.549203 13573 consensus_queue.cc:1048] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.555189 13569 mvcc.cc:204] Tried to move back new op lower bound from 7248418801856741376 to 7248418800329986048. Current Snapshot: MvccSnapshot[applied={T|T < 7248418801856741376}]
I20260128 20:51:36.575223 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.571830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:51145 (local address 127.4.183.129:39527)
0128 20:51:36.571972 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.571976 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.572090 (+ 114us) server_negotiation.cc:408] Connection header received
0128 20:51:36.572253 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.572257 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.572329 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.572418 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.572932 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.573508 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.574467 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.574684 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.574730 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.574813 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.574904 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.575018 (+ 114us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.575071 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 20:51:36.575241 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.571612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:39527 (local address 127.4.183.131:51145)
0128 20:51:36.571998 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.572014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.572139 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.572436 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.572440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.572448 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:51:36.572780 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.572789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.573747 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.573752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.574333 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.574343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.574485 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.574545 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.574912 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.574920 (+ 8us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.575007 (+ 87us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.575095 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":141,"threads_started":1}
I20260128 20:51:36.602392 13037 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:36.602649 13037 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:36.605226 13037 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Sending DeleteTablet for 3 replicas of tablet 00aeeda83f5242d68d0a8bd66f0eae62
I20260128 20:51:36.606026 13318 tablet_service.cc:1552] Processing DeleteTablet for tablet 00aeeda83f5242d68d0a8bd66f0eae62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:33716
I20260128 20:51:36.606034 13460 tablet_service.cc:1552] Processing DeleteTablet for tablet 00aeeda83f5242d68d0a8bd66f0eae62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:40270
I20260128 20:51:36.606379 13172 tablet_service.cc:1552] Processing DeleteTablet for tablet 00aeeda83f5242d68d0a8bd66f0eae62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:44600
I20260128 20:51:36.606442 13613 tablet_replica.cc:333] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: stopping tablet replica
I20260128 20:51:36.606570 13613 raft_consensus.cc:2243] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:36.606721 13613 raft_consensus.cc:2272] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.607120 13612 tablet_replica.cc:333] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: stopping tablet replica
I20260128 20:51:36.607112 13614 tablet_replica.cc:333] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: stopping tablet replica
I20260128 20:51:36.607254 13612 raft_consensus.cc:2243] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:36.607367 13614 raft_consensus.cc:2243] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:36.607385 13612 raft_consensus.cc:2272] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.607659 13614 raft_consensus.cc:2272] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.608245 13612 ts_tablet_manager.cc:1916] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.608477 13614 ts_tablet_manager.cc:1916] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.611452 13612 ts_tablet_manager.cc:1929] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:51:36.611452 13614 ts_tablet_manager.cc:1929] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:51:36.611572 13614 log.cc:1199] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/00aeeda83f5242d68d0a8bd66f0eae62
I20260128 20:51:36.611582 13612 log.cc:1199] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/00aeeda83f5242d68d0a8bd66f0eae62
I20260128 20:51:36.611874 13612 ts_tablet_manager.cc:1950] T 00aeeda83f5242d68d0a8bd66f0eae62 P e135aeabf5894f10950acf03519e9730: Deleting consensus metadata
I20260128 20:51:36.611940 13614 ts_tablet_manager.cc:1950] T 00aeeda83f5242d68d0a8bd66f0eae62 P 7f6e83fc466c4e169f49e61dda87d042: Deleting consensus metadata
I20260128 20:51:36.612896 13023 catalog_manager.cc:4985] TS e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477): tablet 00aeeda83f5242d68d0a8bd66f0eae62 (table test-table [id=c6f5b7a4a3264dba8b096d15f3381513]) successfully deleted
I20260128 20:51:36.612987 13022 catalog_manager.cc:4985] TS 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527): tablet 00aeeda83f5242d68d0a8bd66f0eae62 (table test-table [id=c6f5b7a4a3264dba8b096d15f3381513]) successfully deleted
I20260128 20:51:36.613688 13613 ts_tablet_manager.cc:1916] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.617007 13613 ts_tablet_manager.cc:1929] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:51:36.617125 13613 log.cc:1199] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/00aeeda83f5242d68d0a8bd66f0eae62
I20260128 20:51:36.617448 13613 ts_tablet_manager.cc:1950] T 00aeeda83f5242d68d0a8bd66f0eae62 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting consensus metadata
I20260128 20:51:36.618391 13022 catalog_manager.cc:4985] TS 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403): tablet 00aeeda83f5242d68d0a8bd66f0eae62 (table test-table [id=c6f5b7a4a3264dba8b096d15f3381513]) successfully deleted
Jan 28 20:51:36 dist-test-slave-hss7 krb5kdc[12989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 20:51:36 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:36 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 20:51:36.637398 13086 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:36.630307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.4.183.190:34393)
0128 20:51:36.630472 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.630477 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.630644 (+ 167us) server_negotiation.cc:408] Connection header received
0128 20:51:36.630790 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.630794 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.630868 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.630960 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:36.631935 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.632756 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.633708 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.633928 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.637250 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47938: BlockingRecv error: recv got EOF from 127.0.0.1:47938 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jan 28 20:51:36 dist-test-slave-hss7 krb5kdc[12989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM
I20260128 20:51:36.649358 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.639210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.4.183.190:34393)
0128 20:51:36.639357 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.639361 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.639482 (+ 121us) server_negotiation.cc:408] Connection header received
0128 20:51:36.639736 (+ 254us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.639741 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.639846 (+ 105us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.639979 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:36.641261 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.641799 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.642841 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.643128 (+ 287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.645853 (+ 2725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:36.645891 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:36.645895 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:36.645944 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:36.647639 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:36.648181 (+ 542us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:36.648185 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:36.648187 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:36.648245 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:36.648458 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:36.648461 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:36.648463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:36.648661 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:36.648805 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.649010 (+ 205us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.649186 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260128 20:51:36.652313 13037 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:36.652856 13037 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:36.658749 13172 tablet_service.cc:1505] Processing CreateTablet for tablet 29145fdbb8c6423db1268f5a9dd9602e (DEFAULT_TABLE table=test-table [id=218a7fe3ec684a9eb341cbfeb6d77d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.659111 13172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29145fdbb8c6423db1268f5a9dd9602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.659041 13318 tablet_service.cc:1505] Processing CreateTablet for tablet 29145fdbb8c6423db1268f5a9dd9602e (DEFAULT_TABLE table=test-table [id=218a7fe3ec684a9eb341cbfeb6d77d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.659363 13318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29145fdbb8c6423db1268f5a9dd9602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.661409 13556 tablet_bootstrap.cc:492] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Bootstrap starting.
I20260128 20:51:36.662200 13460 tablet_service.cc:1505] Processing CreateTablet for tablet 29145fdbb8c6423db1268f5a9dd9602e (DEFAULT_TABLE table=test-table [id=218a7fe3ec684a9eb341cbfeb6d77d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.662492 13460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29145fdbb8c6423db1268f5a9dd9602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.662706 13556 tablet_bootstrap.cc:654] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.663962 13556 tablet_bootstrap.cc:492] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: No bootstrap required, opened a new log
I20260128 20:51:36.664089 13556 ts_tablet_manager.cc:1403] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:36.664701 13556 raft_consensus.cc:359] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.664834 13556 raft_consensus.cc:385] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.664872 13556 raft_consensus.cc:740] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e135aeabf5894f10950acf03519e9730, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.664996 13554 tablet_bootstrap.cc:492] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Bootstrap starting.
I20260128 20:51:36.665055 13556 consensus_queue.cc:260] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.665524 13556 ts_tablet_manager.cc:1434] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260128 20:51:36.666051 13554 tablet_bootstrap.cc:654] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.667500 13554 tablet_bootstrap.cc:492] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: No bootstrap required, opened a new log
I20260128 20:51:36.667600 13554 ts_tablet_manager.cc:1403] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:36.667616 13555 tablet_bootstrap.cc:492] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Bootstrap starting.
I20260128 20:51:36.668191 13554 raft_consensus.cc:359] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.668313 13554 raft_consensus.cc:385] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.668356 13554 raft_consensus.cc:740] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.668494 13554 consensus_queue.cc:260] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.668684 13555 tablet_bootstrap.cc:654] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.668807 13554 ts_tablet_manager.cc:1434] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 20:51:36.671216 13555 tablet_bootstrap.cc:492] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: No bootstrap required, opened a new log
I20260128 20:51:36.671332 13555 ts_tablet_manager.cc:1403] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260128 20:51:36.671808 13555 raft_consensus.cc:359] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.671952 13555 raft_consensus.cc:385] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.671993 13555 raft_consensus.cc:740] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.672130 13555 consensus_queue.cc:260] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.672397 13555 ts_tablet_manager.cc:1434] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260128 20:51:36.783589 13562 raft_consensus.cc:493] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:36.783759 13562 raft_consensus.cc:515] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.784854 13562 leader_election.cc:290] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527), 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:36.788475 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.785574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:41403 (local address 127.4.183.130:34941)
0128 20:51:36.785875 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.785889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.785992 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.786242 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.786244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.786259 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:36.786467 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.786473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.787313 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.787318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.788009 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.788019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.788159 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.788185 (+ 26us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.788241 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.788309 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260128 20:51:36.788915 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.785679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:34941 (local address 127.4.183.131:41403)
0128 20:51:36.785840 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.785844 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.785961 (+ 117us) server_negotiation.cc:408] Connection header received
0128 20:51:36.786082 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.786085 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.786134 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.786225 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:36.786578 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.787190 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.788140 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.788621 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.788669 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.788729 (+ 60us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.788803 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 20:51:36.789301 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.785286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:56423 (local address 127.4.183.129:39527)
0128 20:51:36.785429 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.785434 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.785609 (+ 175us) server_negotiation.cc:408] Connection header received
0128 20:51:36.785876 (+ 267us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.785882 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.785954 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.786039 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:51:36.786837 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.787364 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.788348 (+ 984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.788801 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.788841 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.789077 (+ 236us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.789185 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 20:51:36.789386 13480 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29145fdbb8c6423db1268f5a9dd9602e" candidate_uuid: "e135aeabf5894f10950acf03519e9730" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" is_pre_election: true
I20260128 20:51:36.789535 13480 raft_consensus.cc:2468] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e135aeabf5894f10950acf03519e9730 in term 0.
I20260128 20:51:36.789733 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.785201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:39527 (local address 127.4.183.130:56423)
0128 20:51:36.785510 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.785527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.785677 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.786061 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.786064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.786084 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:51:36.786697 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.786708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.787477 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.787480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.788194 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.788204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.788911 (+ 707us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.788941 (+ 30us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.789520 (+ 579us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.789596 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260128 20:51:36.790019 13270 leader_election.cc:304] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, e135aeabf5894f10950acf03519e9730; no voters:
I20260128 20:51:36.790560 13562 raft_consensus.cc:2804] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:36.790618 13562 raft_consensus.cc:493] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:36.790654 13562 raft_consensus.cc:3060] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.790637 13196 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29145fdbb8c6423db1268f5a9dd9602e" candidate_uuid: "e135aeabf5894f10950acf03519e9730" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e83fc466c4e169f49e61dda87d042" is_pre_election: true
I20260128 20:51:36.790789 13196 raft_consensus.cc:2468] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e135aeabf5894f10950acf03519e9730 in term 0.
I20260128 20:51:36.792021 13562 raft_consensus.cc:515] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.792449 13562 leader_election.cc:290] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527), 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403)
I20260128 20:51:36.792954 13196 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29145fdbb8c6423db1268f5a9dd9602e" candidate_uuid: "e135aeabf5894f10950acf03519e9730" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e83fc466c4e169f49e61dda87d042"
I20260128 20:51:36.793084 13196 raft_consensus.cc:3060] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.794306 13196 raft_consensus.cc:2468] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e135aeabf5894f10950acf03519e9730 in term 1.
I20260128 20:51:36.794607 13480 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29145fdbb8c6423db1268f5a9dd9602e" candidate_uuid: "e135aeabf5894f10950acf03519e9730" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2"
I20260128 20:51:36.794682 13270 leader_election.cc:304] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6e83fc466c4e169f49e61dda87d042, e135aeabf5894f10950acf03519e9730; no voters:
I20260128 20:51:36.794730 13480 raft_consensus.cc:3060] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.794922 13562 raft_consensus.cc:2804] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:36.795159 13562 raft_consensus.cc:697] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 1 LEADER]: Becoming Leader. State: Replica: e135aeabf5894f10950acf03519e9730, State: Running, Role: LEADER
I20260128 20:51:36.795486 13562 consensus_queue.cc:237] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.795769 13480 raft_consensus.cc:2468] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e135aeabf5894f10950acf03519e9730 in term 1.
I20260128 20:51:36.798424 13035 catalog_manager.cc:5654] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 reported cstate change: term changed from 0 to 1, leader changed from <none> to e135aeabf5894f10950acf03519e9730 (127.4.183.130). New cstate: current_term: 1 leader_uuid: "e135aeabf5894f10950acf03519e9730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } health_report { overall_health: UNKNOWN } } }
I20260128 20:51:36.829229 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.826002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.4.183.130:40477)
0128 20:51:36.826147 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.826151 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.826168 (+ 17us) server_negotiation.cc:408] Connection header received
0128 20:51:36.826257 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.826259 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.826317 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.826405 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.826750 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.827222 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.827947 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.828089 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.828146 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.828687 (+ 541us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.828787 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.828920 (+ 133us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.829016 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260128 20:51:36.832291 13196 raft_consensus.cc:1275] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Refusing update from remote peer e135aeabf5894f10950acf03519e9730: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.832325 13480 raft_consensus.cc:1275] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Refusing update from remote peer e135aeabf5894f10950acf03519e9730: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.832851 13562 consensus_queue.cc:1048] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.833082 13632 consensus_queue.cc:1048] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.846619 13035 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:36.846827 13035 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:36.849485 13035 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Sending DeleteTablet for 3 replicas of tablet 29145fdbb8c6423db1268f5a9dd9602e
I20260128 20:51:36.850314 13318 tablet_service.cc:1552] Processing DeleteTablet for tablet 29145fdbb8c6423db1268f5a9dd9602e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:33716
I20260128 20:51:36.850504 13172 tablet_service.cc:1552] Processing DeleteTablet for tablet 29145fdbb8c6423db1268f5a9dd9602e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:44600
I20260128 20:51:36.850497 13460 tablet_service.cc:1552] Processing DeleteTablet for tablet 29145fdbb8c6423db1268f5a9dd9602e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:36 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:40270
I20260128 20:51:36.850579 13612 tablet_replica.cc:333] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: stopping tablet replica
I20260128 20:51:36.850703 13614 tablet_replica.cc:333] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: stopping tablet replica
I20260128 20:51:36.850709 13612 raft_consensus.cc:2243] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:36.850848 13614 raft_consensus.cc:2243] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:36.850971 13614 raft_consensus.cc:2272] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.851075 13612 raft_consensus.cc:2272] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.851158 13613 tablet_replica.cc:333] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: stopping tablet replica
I20260128 20:51:36.851274 13613 raft_consensus.cc:2243] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:36.851411 13613 raft_consensus.cc:2272] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:36.851917 13612 ts_tablet_manager.cc:1916] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.852803 13614 ts_tablet_manager.cc:1916] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.852777 13035 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:51:36.853650 13035 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:51:36.854668 13612 ts_tablet_manager.cc:1929] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:36.854781 13612 log.cc:1199] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/29145fdbb8c6423db1268f5a9dd9602e
I20260128 20:51:36.855079 13612 ts_tablet_manager.cc:1950] T 29145fdbb8c6423db1268f5a9dd9602e P e135aeabf5894f10950acf03519e9730: Deleting consensus metadata
I20260128 20:51:36.855572 13614 ts_tablet_manager.cc:1929] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:36.855662 13614 log.cc:1199] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-0/wal/wals/29145fdbb8c6423db1268f5a9dd9602e
I20260128 20:51:36.855944 13614 ts_tablet_manager.cc:1950] T 29145fdbb8c6423db1268f5a9dd9602e P 7f6e83fc466c4e169f49e61dda87d042: Deleting consensus metadata
I20260128 20:51:36.856684 13022 catalog_manager.cc:4985] TS 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527): tablet 29145fdbb8c6423db1268f5a9dd9602e (table test-table [id=218a7fe3ec684a9eb341cbfeb6d77d11]) successfully deleted
I20260128 20:51:36.858040 13023 catalog_manager.cc:4985] TS e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477): tablet 29145fdbb8c6423db1268f5a9dd9602e (table test-table [id=218a7fe3ec684a9eb341cbfeb6d77d11]) successfully deleted
I20260128 20:51:36.858945 13613 ts_tablet_manager.cc:1916] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:36.860483 13172 tablet_service.cc:1505] Processing CreateTablet for tablet d61dd3b390eb4491a0b6b25770fc8d87 (DEFAULT_TABLE table=test-table [id=b19b5676314c4ba09bb0dcc22a153e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.860805 13172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61dd3b390eb4491a0b6b25770fc8d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.861614 13613 ts_tablet_manager.cc:1929] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:51:36.861685 13613 log.cc:1199] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/29145fdbb8c6423db1268f5a9dd9602e
I20260128 20:51:36.861889 13613 ts_tablet_manager.cc:1950] T 29145fdbb8c6423db1268f5a9dd9602e P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting consensus metadata
I20260128 20:51:36.863520 13555 tablet_bootstrap.cc:492] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042: Bootstrap starting.
I20260128 20:51:36.863750 13318 tablet_service.cc:1505] Processing CreateTablet for tablet d61dd3b390eb4491a0b6b25770fc8d87 (DEFAULT_TABLE table=test-table [id=b19b5676314c4ba09bb0dcc22a153e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.864094 13318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61dd3b390eb4491a0b6b25770fc8d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.864542 13555 tablet_bootstrap.cc:654] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.866127 13555 tablet_bootstrap.cc:492] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042: No bootstrap required, opened a new log
I20260128 20:51:36.866250 13555 ts_tablet_manager.cc:1403] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:36.866652 13556 tablet_bootstrap.cc:492] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Bootstrap starting.
I20260128 20:51:36.866761 13555 raft_consensus.cc:359] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.866899 13555 raft_consensus.cc:385] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.866940 13555 raft_consensus.cc:740] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e83fc466c4e169f49e61dda87d042, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.867091 13555 consensus_queue.cc:260] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.867370 13555 ts_tablet_manager.cc:1434] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:36.867765 13556 tablet_bootstrap.cc:654] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.868901 13460 tablet_service.cc:1505] Processing CreateTablet for tablet d61dd3b390eb4491a0b6b25770fc8d87 (DEFAULT_TABLE table=test-table [id=b19b5676314c4ba09bb0dcc22a153e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:51:36.869285 13460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61dd3b390eb4491a0b6b25770fc8d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:36.869956 13556 tablet_bootstrap.cc:492] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: No bootstrap required, opened a new log
I20260128 20:51:36.870047 13556 ts_tablet_manager.cc:1403] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:36.870544 13556 raft_consensus.cc:359] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.870658 13556 raft_consensus.cc:385] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.870692 13556 raft_consensus.cc:740] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e135aeabf5894f10950acf03519e9730, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.870820 13556 consensus_queue.cc:260] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.871119 13556 ts_tablet_manager.cc:1434] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 20:51:36.871907 13554 tablet_bootstrap.cc:492] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Bootstrap starting.
I20260128 20:51:36.872857 13554 tablet_bootstrap.cc:654] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:36.873687 13022 catalog_manager.cc:4985] TS 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403): tablet 29145fdbb8c6423db1268f5a9dd9602e (table test-table [id=218a7fe3ec684a9eb341cbfeb6d77d11]) successfully deleted
I20260128 20:51:36.874995 13554 tablet_bootstrap.cc:492] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: No bootstrap required, opened a new log
I20260128 20:51:36.875100 13554 ts_tablet_manager.cc:1403] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260128 20:51:36.875590 13554 raft_consensus.cc:359] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.875720 13554 raft_consensus.cc:385] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:36.875753 13554 raft_consensus.cc:740] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Initialized, Role: FOLLOWER
I20260128 20:51:36.875885 13554 consensus_queue.cc:260] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.876168 13554 ts_tablet_manager.cc:1434] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 20:51:36.921229 13596 raft_consensus.cc:493] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:51:36.921406 13596 raft_consensus.cc:515] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.921996 13596 leader_election.cc:290] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527), e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477)
I20260128 20:51:36.922528 13196 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61dd3b390eb4491a0b6b25770fc8d87" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e83fc466c4e169f49e61dda87d042" is_pre_election: true
I20260128 20:51:36.922536 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61dd3b390eb4491a0b6b25770fc8d87" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730" is_pre_election: true
I20260128 20:51:36.922696 13196 raft_consensus.cc:2468] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 0.
I20260128 20:51:36.922717 13338 raft_consensus.cc:2468] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 0.
I20260128 20:51:36.923075 13412 leader_election.cc:304] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, 7f6e83fc466c4e169f49e61dda87d042; no voters:
I20260128 20:51:36.923326 13596 raft_consensus.cc:2804] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:51:36.923400 13596 raft_consensus.cc:493] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:51:36.923466 13596 raft_consensus.cc:3060] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.924605 13596 raft_consensus.cc:515] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.925092 13596 leader_election.cc:290] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527), e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477)
I20260128 20:51:36.925593 13338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61dd3b390eb4491a0b6b25770fc8d87" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e135aeabf5894f10950acf03519e9730"
I20260128 20:51:36.925594 13196 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61dd3b390eb4491a0b6b25770fc8d87" candidate_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e83fc466c4e169f49e61dda87d042"
I20260128 20:51:36.925745 13196 raft_consensus.cc:3060] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.925748 13338 raft_consensus.cc:3060] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:36.926890 13338 raft_consensus.cc:2468] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 1.
I20260128 20:51:36.926916 13196 raft_consensus.cc:2468] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0aaa6f423c4e69aab2f9b662cf78e2 in term 1.
I20260128 20:51:36.927356 13413 leader_election.cc:304] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a0aaa6f423c4e69aab2f9b662cf78e2, 7f6e83fc466c4e169f49e61dda87d042, e135aeabf5894f10950acf03519e9730; no voters:
I20260128 20:51:36.927656 13596 raft_consensus.cc:2804] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:36.927755 13596 raft_consensus.cc:697] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 LEADER]: Becoming Leader. State: Replica: 3a0aaa6f423c4e69aab2f9b662cf78e2, State: Running, Role: LEADER
I20260128 20:51:36.927904 13596 consensus_queue.cc:237] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } }
I20260128 20:51:36.929970 13035 catalog_manager.cc:5654] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131). New cstate: current_term: 1 leader_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a0aaa6f423c4e69aab2f9b662cf78e2" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 41403 } health_report { overall_health: HEALTHY } } }
I20260128 20:51:36.961246 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.958292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.4.183.131:41403)
0128 20:51:36.958477 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.958482 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.958501 (+ 19us) server_negotiation.cc:408] Connection header received
0128 20:51:36.958548 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.958551 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.958601 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.958693 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.959075 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.959539 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.960279 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.960469 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.960514 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.960599 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.960701 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.960820 (+ 119us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.960876 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260128 20:51:36.972131 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.968915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:41403 (local address 127.4.183.131:48925)
0128 20:51:36.969177 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:36.969194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:36.969293 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:36.969556 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:36.969558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:36.969566 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 20:51:36.969745 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.969750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.970772 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.970775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:36.971213 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:36.971222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.971341 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.971373 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.971827 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.971846 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:51:36.971910 (+ 64us) client_negotiation.cc:240] Negotiation successful
0128 20:51:36.971986 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260128 20:51:36.972134 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:36.968996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:48925 (local address 127.4.183.131:41403)
0128 20:51:36.969174 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:36.969179 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:36.969261 (+ 82us) server_negotiation.cc:408] Connection header received
0128 20:51:36.969383 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:36.969387 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:36.969447 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:36.969542 (+ 95us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:51:36.969849 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.970597 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:36.971346 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:36.971552 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:36.971601 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:51:36.971707 (+ 106us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:51:36.971792 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:36.971942 (+ 150us) server_negotiation.cc:300] Negotiation successful
0128 20:51:36.972001 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 20:51:36.979337 13196 raft_consensus.cc:1275] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Refusing update from remote peer 3a0aaa6f423c4e69aab2f9b662cf78e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.979337 13338 raft_consensus.cc:1275] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Refusing update from remote peer 3a0aaa6f423c4e69aab2f9b662cf78e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:51:36.979871 13596 consensus_queue.cc:1048] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6e83fc466c4e169f49e61dda87d042" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:36.980126 13603 consensus_queue.cc:1048] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e135aeabf5894f10950acf03519e9730" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:51:37.029732 13035 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:37.029940 13035 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:51:37.032672 13035 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 60c6f7178eb04cacb9ef0a1ef6fa009d: Sending DeleteTablet for 3 replicas of tablet d61dd3b390eb4491a0b6b25770fc8d87
I20260128 20:51:37.033694 13172 tablet_service.cc:1552] Processing DeleteTablet for tablet d61dd3b390eb4491a0b6b25770fc8d87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:37 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:44600
I20260128 20:51:37.033922 13318 tablet_service.cc:1552] Processing DeleteTablet for tablet d61dd3b390eb4491a0b6b25770fc8d87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:37 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:33716
I20260128 20:51:37.034130 13614 tablet_replica.cc:333] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042: stopping tablet replica
I20260128 20:51:37.034152 13612 tablet_replica.cc:333] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: stopping tablet replica
I20260128 20:51:37.034255 13614 raft_consensus.cc:2243] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:37.034276 13612 raft_consensus.cc:2243] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:51:37.034410 13612 raft_consensus.cc:2272] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:37.034397 13614 raft_consensus.cc:2272] T d61dd3b390eb4491a0b6b25770fc8d87 P 7f6e83fc466c4e169f49e61dda87d042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:37.035291 13612 ts_tablet_manager.cc:1916] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:37.036194 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13111
I20260128 20:51:37.038435 13612 ts_tablet_manager.cc:1929] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:51:37.038463 13460 tablet_service.cc:1552] Processing DeleteTablet for tablet d61dd3b390eb4491a0b6b25770fc8d87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:51:37 UTC) from {username='oryx', principal='oryx/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:40270
I20260128 20:51:37.038532 13612 log.cc:1199] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/d61dd3b390eb4491a0b6b25770fc8d87
I20260128 20:51:37.038815 13613 tablet_replica.cc:333] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: stopping tablet replica
I20260128 20:51:37.038839 13612 ts_tablet_manager.cc:1950] T d61dd3b390eb4491a0b6b25770fc8d87 P e135aeabf5894f10950acf03519e9730: Deleting consensus metadata
I20260128 20:51:37.038949 13613 raft_consensus.cc:2243] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:51:37.039304 13613 raft_consensus.cc:2272] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:51:37.039680 13023 catalog_manager.cc:4985] TS e135aeabf5894f10950acf03519e9730 (127.4.183.130:40477): tablet d61dd3b390eb4491a0b6b25770fc8d87 (table test-table [id=b19b5676314c4ba09bb0dcc22a153e52]) successfully deleted
I20260128 20:51:37.040222 13613 ts_tablet_manager.cc:1916] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:51:37.043641 13613 ts_tablet_manager.cc:1929] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 20:51:37.043766 13613 log.cc:1199] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/d61dd3b390eb4491a0b6b25770fc8d87
I20260128 20:51:37.044049 13613 ts_tablet_manager.cc:1950] T d61dd3b390eb4491a0b6b25770fc8d87 P 3a0aaa6f423c4e69aab2f9b662cf78e2: Deleting consensus metadata
I20260128 20:51:37.045116 13022 catalog_manager.cc:4985] TS 3a0aaa6f423c4e69aab2f9b662cf78e2 (127.4.183.131:41403): tablet d61dd3b390eb4491a0b6b25770fc8d87 (table test-table [id=b19b5676314c4ba09bb0dcc22a153e52]) successfully deleted
I20260128 20:51:37.048079 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13253
W20260128 20:51:37.049351 13022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.183.129:39527 (error 108)
I20260128 20:51:37.051390 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:37.050818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:39527 (local address 127.0.0.1:44618)
0128 20:51:37.051150 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:37.051239 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.129:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
W20260128 20:51:37.051602 13022 catalog_manager.cc:4712] TS 7f6e83fc466c4e169f49e61dda87d042 (127.4.183.129:39527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d61dd3b390eb4491a0b6b25770fc8d87: Network error: Client connection negotiation failed: client connection to 127.4.183.129:39527: connect: Connection refused (error 111)
I20260128 20:51:37.055975 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13395
I20260128 20:51:37.063620 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13005
2026-01-28T20:51:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4599 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:51:37 dist-test-slave-hss7 krb5kdc[13652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:51:37 dist-test-slave-hss7 krb5kdc[13652](info): set up 2 sockets
Jan 28 20:51:37 dist-test-slave-hss7 krb5kdc[13652](info): commencing operation
krb5kdc: starting...
W20260128 20:51:39.095362 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.006s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:51:39 dist-test-slave-hss7 krb5kdc[13652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:51:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:39.256595 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:44941
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:44941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:39.377341 13668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:39.377684 13668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:39.377806 13668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:39.382536 13668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:39.382695 13668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:39.382740 13668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:39.382786 13668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:39.382855 13668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:39.389447 13668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:44941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:44941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:39.390852 13668 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:39.391877 13668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:39.402319 13674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:39.402396 13673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:39.402319 13676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:39.403208 13668 server_base.cc:1060] running on GCE node
I20260128 20:51:39.403887 13668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:39.405277 13668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:39.406518 13668 hybrid_clock.cc:648] HybridClock initialized: now 1769633499406506 us; error 41 us; skew 500 ppm
W20260128 20:51:39.410838 13668 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-01-28T20:51:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2350 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:51:39 dist-test-slave-hss7 krb5kdc[13681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:51:39 dist-test-slave-hss7 krb5kdc[13681](info): set up 2 sockets
Jan 28 20:51:39 dist-test-slave-hss7 krb5kdc[13681](info): commencing operation
krb5kdc: starting...
W20260128 20:51:41.471340 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:51:41 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:51:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:41.632602 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39755
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:41.756525 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:41.756881 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:41.756981 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:41.760854 13697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:41.760993 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:41.761037 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:41.761080 13697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:41.761125 13697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:41.766211 13697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:41.767589 13697 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:41.768581 13697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:41.775676 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:41.775679 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:41.775933 13697 server_base.cc:1060] running on GCE node
W20260128 20:51:41.775679 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:41.776698 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:41.777868 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:41.779071 13697 hybrid_clock.cc:648] HybridClock initialized: now 1769633501779062 us; error 55 us; skew 500 ppm
Jan 28 20:51:41 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633501, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:41.782346 13697 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:51:41.783862 13697 webserver.cc:492] Webserver started at http://127.4.183.190:33293/ using document root <none> and password file <none>
I20260128 20:51:41.784554 13697 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:41.784636 13697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:41.784894 13697 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:41.786939 13697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "efd72293ccf04103aea5f680d5c4388a"
format_stamp: "Formatted at 2026-01-28 20:51:41 on dist-test-slave-hss7"
server_key: "b5ce4d516075dd15c934fdd1f2e9f7b4"
server_key_iv: "1e934ff3af63601eaa18bab0314a4196"
server_key_version: "encryptionkey@0"
I20260128 20:51:41.787528 13697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "efd72293ccf04103aea5f680d5c4388a"
format_stamp: "Formatted at 2026-01-28 20:51:41 on dist-test-slave-hss7"
server_key: "b5ce4d516075dd15c934fdd1f2e9f7b4"
server_key_iv: "1e934ff3af63601eaa18bab0314a4196"
server_key_version: "encryptionkey@0"
I20260128 20:51:41.791702 13697 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:51:41.794636 13712 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:41.795970 13697 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:41.796164 13697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "efd72293ccf04103aea5f680d5c4388a"
format_stamp: "Formatted at 2026-01-28 20:51:41 on dist-test-slave-hss7"
server_key: "b5ce4d516075dd15c934fdd1f2e9f7b4"
server_key_iv: "1e934ff3af63601eaa18bab0314a4196"
server_key_version: "encryptionkey@0"
I20260128 20:51:41.796307 13697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:41.819658 13697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:41.824067 13697 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:41.824290 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:41.832826 13697 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:39755
I20260128 20:51:41.832825 13764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:39755 every 8 connection(s)
I20260128 20:51:41.834187 13697 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:41.837802 13765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:41.840579 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13697
I20260128 20:51:41.840726 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:41.841002 4830 external_mini_cluster.cc:1468] Setting key 9fe4677b4a5ff73fe31ed7fbd8c3dd9e
I20260128 20:51:41.845326 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Bootstrap starting.
I20260128 20:51:41.847995 13765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:41.848879 13765 log.cc:826] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Log is configured to *not* fsync() on all Append() calls
Jan 28 20:51:41 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:41.852056 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: No bootstrap required, opened a new log
I20260128 20:51:41.855350 13765 raft_consensus.cc:359] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:41.855577 13765 raft_consensus.cc:385] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:41.855654 13765 raft_consensus.cc:740] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd72293ccf04103aea5f680d5c4388a, State: Initialized, Role: FOLLOWER
I20260128 20:51:41.856166 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:41.842601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.4.183.190:39755)
0128 20:51:41.843161 (+ 560us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:41.843174 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:41.843213 (+ 39us) server_negotiation.cc:408] Connection header received
0128 20:51:41.844053 (+ 840us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:41.844080 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:41.844471 (+ 391us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:41.844956 (+ 485us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:41.846064 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:41.846840 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:41.847584 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:41.847970 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:41.850925 (+ 2955us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:41.850949 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:41.850962 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:41.851006 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:41.853020 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:41.853620 (+ 600us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:41.853626 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:41.853630 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:41.853745 (+ 115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:41.854046 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:41.854050 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:41.854051 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:41.854461 (+ 410us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:41.854681 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:41.854971 (+ 290us) server_negotiation.cc:300] Negotiation successful
0128 20:51:41.855326 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":183,"threads_started":1}
I20260128 20:51:41.856160 13765 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:41.856573 13765 raft_consensus.cc:399] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:41.856747 13765 raft_consensus.cc:493] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:41.857015 13765 raft_consensus.cc:3060] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:41.858173 13765 raft_consensus.cc:515] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:41.858594 13765 leader_election.cc:304] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd72293ccf04103aea5f680d5c4388a; no voters:
I20260128 20:51:41.859015 13765 leader_election.cc:290] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:41.859161 13770 raft_consensus.cc:2804] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:41.859376 13770 raft_consensus.cc:697] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 LEADER]: Becoming Leader. State: Replica: efd72293ccf04103aea5f680d5c4388a, State: Running, Role: LEADER
I20260128 20:51:41.859817 13770 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:41.860328 13765 sys_catalog.cc:565] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:41.861604 13771 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd72293ccf04103aea5f680d5c4388a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } } }
I20260128 20:51:41.861963 13771 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:41.861858 13772 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd72293ccf04103aea5f680d5c4388a. Latest consensus state: current_term: 1 leader_uuid: "efd72293ccf04103aea5f680d5c4388a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } } }
I20260128 20:51:41.862329 13772 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: This master's current role is: LEADER
W20260128 20:51:41.865778 13785 catalog_manager.cc:1568] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 20:51:41.865886 13785 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 20:51:41.865975 13779 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:41.866762 13779 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:41.872433 13779 catalog_manager.cc:1357] Generated new cluster ID: de16155012db40eabd13449d9361b7e6
I20260128 20:51:41.872531 13779 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:41.883041 13779 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:41.884439 13779 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:41.893596 13779 catalog_manager.cc:6027] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Generated new TSK 0
I20260128 20:51:41.894623 13779 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:41.957605 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39755
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:42.083760 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:42.084033 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:42.084162 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:42.088155 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:42.088270 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:42.088359 13793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:42.093497 13793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:42.094825 13793 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:42.095981 13793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:42.105217 13801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.105216 13798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.105217 13799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:42.105530 13793 server_base.cc:1060] running on GCE node
I20260128 20:51:42.106128 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:42.106854 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:42.108105 13793 hybrid_clock.cc:648] HybridClock initialized: now 1769633502108067 us; error 75 us; skew 500 ppm
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:42.112334 13793 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:51:42.114190 13793 webserver.cc:492] Webserver started at http://127.4.183.129:39033/ using document root <none> and password file <none>
I20260128 20:51:42.115113 13793 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:42.115203 13793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:42.115396 13793 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:42.117895 13793 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "a407820e4dde46e5b822d3ff7a0e44fc"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "0d8d58abefc4cd69d2ac120b1e11356b"
server_key_iv: "c2020b7fea9147a86cb54d47a0dc153d"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.118435 13793 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "a407820e4dde46e5b822d3ff7a0e44fc"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "0d8d58abefc4cd69d2ac120b1e11356b"
server_key_iv: "c2020b7fea9147a86cb54d47a0dc153d"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.123026 13793 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260128 20:51:42.126670 13808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.128202 13793 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260128 20:51:42.128360 13793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "a407820e4dde46e5b822d3ff7a0e44fc"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "0d8d58abefc4cd69d2ac120b1e11356b"
server_key_iv: "c2020b7fea9147a86cb54d47a0dc153d"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.128475 13793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:42.150249 13793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:42.154713 13793 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:42.154994 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:42.155946 13793 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:42.157001 13793 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:42.157058 13793 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.157104 13793 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:42.157119 13793 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.170277 13793 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:36357
I20260128 20:51:42.170310 13921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:36357 every 8 connection(s)
I20260128 20:51:42.171305 13793 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:42.175881 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13793
I20260128 20:51:42.175998 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:42.176275 4830 external_mini_cluster.cc:1468] Setting key 27a77281c5eee743f8863821343b1f41
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
I20260128 20:51:42.187273 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:42.173314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:60245 (local address 127.4.183.190:39755)
0128 20:51:42.173471 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:42.173475 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:42.174288 (+ 813us) server_negotiation.cc:408] Connection header received
0128 20:51:42.175246 (+ 958us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:42.175251 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:42.175313 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:42.175417 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:42.177645 (+ 2228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.178173 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:42.178861 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.179067 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:42.182142 (+ 3075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:42.182169 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:42.182172 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:42.182205 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:42.184564 (+ 2359us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:42.185491 (+ 927us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:42.185495 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:42.185496 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:42.185555 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:42.185959 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:42.185962 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:42.185963 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:42.186143 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:42.186228 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:42.186982 (+ 754us) server_negotiation.cc:300] Negotiation successful
0128 20:51:42.187140 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 20:51:42.187909 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:42.173590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:60245)
0128 20:51:42.174116 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:42.174160 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:42.175037 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:42.175592 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:42.175601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:42.176153 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:42.177479 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:42.177500 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.178304 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:42.178312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:42.178734 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:42.178742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.179011 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:42.179750 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:42.179799 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:42.181881 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:42.184795 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:42.184807 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:42.184825 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:42.185336 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:42.185662 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:42.185668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:42.185670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:42.185839 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:42.186250 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:42.186257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:42.186541 (+ 284us) client_negotiation.cc:769] Sending connection context
0128 20:51:42.186747 (+ 206us) client_negotiation.cc:240] Negotiation successful
0128 20:51:42.187090 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:51:42.189584 13922 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39755
I20260128 20:51:42.189915 13922 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:42.190570 13922 heartbeater.cc:507] Master 127.4.183.190:39755 requested a full tablet report, sending...
I20260128 20:51:42.192404 13723 ts_manager.cc:194] Registered new tserver with Master: a407820e4dde46e5b822d3ff7a0e44fc (127.4.183.129:36357)
I20260128 20:51:42.194340 13723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:60245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:42.236806 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39755
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:42.358850 13929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:42.359184 13929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:42.359282 13929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:42.363155 13929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:42.363271 13929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:42.363405 13929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:42.368287 13929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:42.369719 13929 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:42.370693 13929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:42.379976 13937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.379854 13935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.379815 13934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:42.381175 13929 server_base.cc:1060] running on GCE node
I20260128 20:51:42.381747 13929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:42.382560 13929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:42.383754 13929 hybrid_clock.cc:648] HybridClock initialized: now 1769633502383726 us; error 57 us; skew 500 ppm
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:42.387491 13929 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:51:42.388931 13929 webserver.cc:492] Webserver started at http://127.4.183.130:44405/ using document root <none> and password file <none>
I20260128 20:51:42.389644 13929 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:42.389707 13929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:42.389935 13929 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:42.392455 13929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "22cf719fc0124ce9884e8fe52da7d612"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "e80a599474d7f5b5d1307d4c39713c7a"
server_key_iv: "ae08966da4721de7222832b86390e6f4"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.393128 13929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "22cf719fc0124ce9884e8fe52da7d612"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "e80a599474d7f5b5d1307d4c39713c7a"
server_key_iv: "ae08966da4721de7222832b86390e6f4"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.397372 13929 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:42.400177 13944 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.401950 13929 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:42.402154 13929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "22cf719fc0124ce9884e8fe52da7d612"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "e80a599474d7f5b5d1307d4c39713c7a"
server_key_iv: "ae08966da4721de7222832b86390e6f4"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.402289 13929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:42.423133 13929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:42.427068 13929 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:42.427328 13929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:42.428001 13929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:42.429039 13929 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:42.429118 13929 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.429216 13929 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:42.429260 13929 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.440069 13929 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36309
I20260128 20:51:42.440096 14057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36309 every 8 connection(s)
I20260128 20:51:42.441293 13929 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:42.444938 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 13929
I20260128 20:51:42.445087 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:42.445363 4830 external_mini_cluster.cc:1468] Setting key c22073be5efddf9ffb1a5766135b1650
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:51:42.457604 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:42.443529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:57793 (local address 127.4.183.190:39755)
0128 20:51:42.443680 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:42.443685 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:42.444530 (+ 845us) server_negotiation.cc:408] Connection header received
0128 20:51:42.445709 (+ 1179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:42.445713 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:42.445785 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:42.445899 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:42.448017 (+ 2118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.448544 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:42.449319 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.449516 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:42.452567 (+ 3051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:42.452595 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:42.452605 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:42.452642 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:42.454725 (+ 2083us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:42.455617 (+ 892us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:42.455623 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:42.455626 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:42.455705 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:42.456076 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:42.456080 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:42.456082 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:42.456291 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:42.456438 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:42.457178 (+ 740us) server_negotiation.cc:300] Negotiation successful
0128 20:51:42.457381 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 20:51:42.458348 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:42.443780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:57793)
0128 20:51:42.444350 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:42.444394 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:42.445420 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:42.446116 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:42.446128 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:42.446742 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:42.447814 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:42.447833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.448701 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:42.448705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:42.449162 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:42.449196 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.449479 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:42.450270 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:42.450303 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:42.452295 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:42.455080 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:42.455089 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:42.455111 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:42.455475 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:42.455823 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:42.455826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:42.455829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:42.455959 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:42.456437 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:42.456448 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:42.456831 (+ 383us) client_negotiation.cc:769] Sending connection context
0128 20:51:42.457089 (+ 258us) client_negotiation.cc:240] Negotiation successful
0128 20:51:42.457512 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":147,"threads_started":1}
I20260128 20:51:42.459767 14058 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39755
I20260128 20:51:42.460050 14058 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:42.460647 14058 heartbeater.cc:507] Master 127.4.183.190:39755 requested a full tablet report, sending...
I20260128 20:51:42.462282 13723 ts_manager.cc:194] Registered new tserver with Master: 22cf719fc0124ce9884e8fe52da7d612 (127.4.183.130:36309)
I20260128 20:51:42.463271 13723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:57793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:51:42.504230 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:39755
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:42.628077 14065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:42.628378 14065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:42.628444 14065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:42.632155 14065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:42.632253 14065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:42.632341 14065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:42.637240 14065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:42.638546 14065 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:42.639474 14065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:42.647063 14070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.647063 14071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.647085 14073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:42.647578 14065 server_base.cc:1060] running on GCE node
I20260128 20:51:42.648046 14065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:42.648909 14065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:42.650228 14065 hybrid_clock.cc:648] HybridClock initialized: now 1769633502650218 us; error 85 us; skew 500 ppm
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:42.653755 14065 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:51:42.655208 14065 webserver.cc:492] Webserver started at http://127.4.183.131:33073/ using document root <none> and password file <none>
I20260128 20:51:42.655833 14065 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:42.655900 14065 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:42.656080 14065 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:42.658111 14065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "457e0998265d4023ad3ff570a6c366fa"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "c4ca7ca9484bfe1e1bfd44395c33da2b"
server_key_iv: "433b84a6e10d0c09c038b678fd0ca691"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.658639 14065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "457e0998265d4023ad3ff570a6c366fa"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "c4ca7ca9484bfe1e1bfd44395c33da2b"
server_key_iv: "433b84a6e10d0c09c038b678fd0ca691"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.663282 14065 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260128 20:51:42.666998 14080 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.668494 14065 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:42.668659 14065 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "457e0998265d4023ad3ff570a6c366fa"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "c4ca7ca9484bfe1e1bfd44395c33da2b"
server_key_iv: "433b84a6e10d0c09c038b678fd0ca691"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.668785 14065 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:42.685093 14065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:42.688568 14065 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:42.688752 14065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:42.689395 14065 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:42.690371 14065 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:42.690424 14065 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.690469 14065 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:42.690485 14065 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.702654 14065 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:45819
I20260128 20:51:42.702672 14193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:45819 every 8 connection(s)
I20260128 20:51:42.703809 14065 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:42.712129 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14065
I20260128 20:51:42.712281 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:42.712672 4830 external_mini_cluster.cc:1468] Setting key eee056836261d43431d76e137619f001
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:51:42.719427 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:42.706361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:54183 (local address 127.4.183.190:39755)
0128 20:51:42.706494 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:42.706498 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:42.707430 (+ 932us) server_negotiation.cc:408] Connection header received
0128 20:51:42.708429 (+ 999us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:42.708433 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:42.708498 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:42.708608 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:42.710595 (+ 1987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.711124 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:42.711848 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.712040 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:42.714747 (+ 2707us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:42.714775 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:42.714777 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:42.714809 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:42.716343 (+ 1534us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:42.717389 (+ 1046us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:42.717505 (+ 116us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:42.717507 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:42.717652 (+ 145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:42.718152 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:42.718155 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:42.718156 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:42.718340 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:42.718448 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:42.719071 (+ 623us) server_negotiation.cc:300] Negotiation successful
0128 20:51:42.719241 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260128 20:51:42.720147 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:42.706669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:54183)
0128 20:51:42.707198 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:42.707241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:42.708194 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:42.708787 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:42.708797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:42.709348 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:42.710379 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:42.710398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.711275 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:42.711284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:42.711707 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:42.711716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:42.712022 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:42.712816 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:42.712870 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:42.714534 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:42.716543 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:42.716550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:42.716567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:42.717226 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:42.717828 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:51:42.717832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:51:42.717835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:51:42.718017 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:51:42.718471 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:42.718485 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:42.718777 (+ 292us) client_negotiation.cc:769] Sending connection context
0128 20:51:42.719017 (+ 240us) client_negotiation.cc:240] Negotiation successful
0128 20:51:42.719342 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":142,"threads_started":1}
I20260128 20:51:42.721704 14194 heartbeater.cc:344] Connected to a master server at 127.4.183.190:39755
I20260128 20:51:42.721975 14194 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:42.722631 14194 heartbeater.cc:507] Master 127.4.183.190:39755 requested a full tablet report, sending...
I20260128 20:51:42.724049 13723 ts_manager.cc:194] Registered new tserver with Master: 457e0998265d4023ad3ff570a6c366fa (127.4.183.131:45819)
I20260128 20:51:42.724730 13723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:54183
I20260128 20:51:42.727777 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.129.keytab.
Entry for principal oryx/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.129.keytab.
WARNING: no policy specified for oryx/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.130.keytab.
Entry for principal oryx/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.130.keytab.
WARNING: no policy specified for oryx/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.131.keytab.
Entry for principal oryx/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.131.keytab.
I20260128 20:51:42.813375 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13793
I20260128 20:51:42.819808 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13929
I20260128 20:51:42.826633 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14065
I20260128 20:51:42.832805 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 13697
I20260128 20:51:42.842060 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39755
--webserver_interface=127.4.183.190
--webserver_port=33293
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:51:42.962512 14203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:42.962836 14203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:42.962971 14203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:42.966557 14203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:42.966696 14203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:42.966743 14203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:42.966804 14203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:42.966857 14203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:42.971737 14203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=33293
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:42.973049 14203 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:42.973994 14203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:42.980466 14209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.980561 14211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:42.980470 14208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:42.980860 14203 server_base.cc:1060] running on GCE node
I20260128 20:51:42.981624 14203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:42.982714 14203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:42.983913 14203 hybrid_clock.cc:648] HybridClock initialized: now 1769633502983906 us; error 42 us; skew 500 ppm
Jan 28 20:51:42 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:42.987294 14203 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:51:42.988663 14203 webserver.cc:492] Webserver started at http://127.4.183.190:33293/ using document root <none> and password file <none>
I20260128 20:51:42.989398 14203 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:42.989504 14203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:42.993670 14203 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:42.996129 14218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:42.997411 14203 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:42.997599 14203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "efd72293ccf04103aea5f680d5c4388a"
format_stamp: "Formatted at 2026-01-28 20:51:41 on dist-test-slave-hss7"
server_key: "b5ce4d516075dd15c934fdd1f2e9f7b4"
server_key_iv: "1e934ff3af63601eaa18bab0314a4196"
server_key_version: "encryptionkey@0"
I20260128 20:51:42.998303 14203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:43.019136 14203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:43.023440 14203 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:43.023695 14203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:43.033012 14270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:39755 every 8 connection(s)
I20260128 20:51:43.033028 14203 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:39755
I20260128 20:51:43.035019 14203 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:43.039781 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14203
I20260128 20:51:43.040329 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:36357
--local_ip_for_outbound_sockets=127.4.183.129
--tserver_master_addrs=127.4.183.190:39755
--webserver_port=39033
--webserver_interface=127.4.183.129
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.183.129
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 20:51:43.040939 14271 sys_catalog.cc:263] Verifying existing consensus state
I20260128 20:51:43.043257 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Bootstrap starting.
I20260128 20:51:43.060384 14271 log.cc:826] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:43.067574 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260128 20:51:43.068053 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Bootstrap complete.
I20260128 20:51:43.071806 14271 raft_consensus.cc:359] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:43.072173 14271 raft_consensus.cc:740] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd72293ccf04103aea5f680d5c4388a, State: Initialized, Role: FOLLOWER
I20260128 20:51:43.072708 14271 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:43.072834 14271 raft_consensus.cc:399] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:43.072925 14271 raft_consensus.cc:493] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:43.073020 14271 raft_consensus.cc:3060] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 1 FOLLOWER]: Advancing to term 2
I20260128 20:51:43.074919 14271 raft_consensus.cc:515] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:43.075331 14271 leader_election.cc:304] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd72293ccf04103aea5f680d5c4388a; no voters:
I20260128 20:51:43.075700 14271 leader_election.cc:290] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260128 20:51:43.075803 14275 raft_consensus.cc:2804] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 2 FOLLOWER]: Leader election won for term 2
I20260128 20:51:43.076321 14275 raft_consensus.cc:697] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [term 2 LEADER]: Becoming Leader. State: Replica: efd72293ccf04103aea5f680d5c4388a, State: Running, Role: LEADER
I20260128 20:51:43.077059 14275 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } }
I20260128 20:51:43.077641 14271 sys_catalog.cc:565] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:43.078933 14277 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd72293ccf04103aea5f680d5c4388a. Latest consensus state: current_term: 2 leader_uuid: "efd72293ccf04103aea5f680d5c4388a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } } }
I20260128 20:51:43.078969 14276 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "efd72293ccf04103aea5f680d5c4388a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd72293ccf04103aea5f680d5c4388a" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 39755 } } }
I20260128 20:51:43.079097 14277 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:43.079098 14276 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:43.079499 14284 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:43.083129 14284 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:43.084017 14284 catalog_manager.cc:1269] Loaded cluster ID: de16155012db40eabd13449d9361b7e6
I20260128 20:51:43.084146 14284 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:43.086990 14284 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:43.088459 14284 catalog_manager.cc:6038] T 00000000000000000000000000000000 P efd72293ccf04103aea5f680d5c4388a: Loaded TSK: 0
I20260128 20:51:43.090055 14284 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260128 20:51:43.169253 14273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:43.169528 14273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:43.169628 14273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:43.173352 14273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:43.173448 14273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:43.173574 14273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:43.178583 14273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.129.keytab
--principal=oryx/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:36357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=39033
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:43.179950 14273 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:43.180927 14273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:43.188380 14300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:43.188369 14297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:43.188366 14298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:43.189064 14273 server_base.cc:1060] running on GCE node
I20260128 20:51:43.189563 14273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:43.190243 14273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:43.191468 14273 hybrid_clock.cc:648] HybridClock initialized: now 1769633503191442 us; error 50 us; skew 500 ppm
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633503, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:43.195328 14273 init.cc:377] Logged in from keytab as oryx/127.4.183.129@KRBTEST.COM (short username oryx)
I20260128 20:51:43.197240 14273 webserver.cc:492] Webserver started at http://127.4.183.129:39033/ using document root <none> and password file <none>
I20260128 20:51:43.197944 14273 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:43.198056 14273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:43.203305 14273 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:51:43.205947 14307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.207204 14273 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:51:43.207365 14273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "a407820e4dde46e5b822d3ff7a0e44fc"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "0d8d58abefc4cd69d2ac120b1e11356b"
server_key_iv: "c2020b7fea9147a86cb54d47a0dc153d"
server_key_version: "encryptionkey@0"
I20260128 20:51:43.207891 14273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:43.234566 14273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:43.238058 14273 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:43.238242 14273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:43.238991 14273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:43.240114 14273 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:43.240172 14273 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.240219 14273 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:43.240273 14273 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.252645 14273 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:36357
I20260128 20:51:43.252660 14420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:36357 every 8 connection(s)
I20260128 20:51:43.253897 14273 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:43.258697 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14273
I20260128 20:51:43.259287 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:36309
--local_ip_for_outbound_sockets=127.4.183.130
--tserver_master_addrs=127.4.183.190:39755
--webserver_port=44405
--webserver_interface=127.4.183.130
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.183.130
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.268824 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.256110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:58511)
0128 20:51:43.256735 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.256789 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.257844 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.259481 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.259494 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.260240 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.262435 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.262494 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.263761 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.263766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.264302 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.264313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.264604 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.265366 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.265398 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.267920 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":134,"threads_started":1}
W20260128 20:51:43.268821 14423 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.256097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:58511 (local address 127.4.183.190:39755)
0128 20:51:43.256674 (+ 577us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.256687 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.256930 (+ 243us) server_negotiation.cc:408] Connection header received
0128 20:51:43.258337 (+ 1407us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.258348 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.258652 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.259006 (+ 354us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.262745 (+ 3739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.263572 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.264440 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.264703 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.268161 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:58511: BlockingRecv error: recv got EOF from 127.4.183.129:58511 (error 108)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":136,"threads_started":1}
W20260128 20:51:43.269801 14421 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:39755 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.183.190:39755: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.275461 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.270310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:35081)
0128 20:51:43.270528 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.270541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.270626 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.270990 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.270993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.271179 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.271372 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.271379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.272142 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.272145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.272525 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.272531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.272670 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.273328 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.273342 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.275296 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260128 20:51:43.275566 14423 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.270401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:35081 (local address 127.4.183.190:39755)
0128 20:51:43.270772 (+ 371us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.270776 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.270787 (+ 11us) server_negotiation.cc:408] Connection header received
0128 20:51:43.270826 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.270828 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.270871 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.270964 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.271517 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.272021 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.272649 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.272868 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.275442 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:35081: BlockingRecv error: recv got EOF from 127.4.183.129:35081 (error 108)
Metrics: {"server-negotiator.queue_time_us":296}
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.281136 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.276159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:56019)
0128 20:51:43.276282 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.276298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.276384 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.276724 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.276727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.276899 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.277074 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.277081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.277878 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.277883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.278263 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.278270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.278404 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.278964 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.278982 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.280975 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260128 20:51:43.281383 14421 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260128 20:51:43.281284 14423 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.276256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:56019 (local address 127.4.183.190:39755)
0128 20:51:43.276515 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.276519 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.276529 (+ 10us) server_negotiation.cc:408] Connection header received
0128 20:51:43.276564 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.276567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.276609 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.276705 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.277244 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.277757 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.278383 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.278979 (+ 596us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.281165 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:56019: BlockingRecv error: recv got EOF from 127.4.183.129:56019 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
W20260128 20:51:43.383813 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:43.384135 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:43.384233 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:43.388227 14425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:43.388350 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:43.388468 14425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:43.393431 14425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.130.keytab
--principal=oryx/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:36309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=44405
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:43.394722 14425 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:43.395668 14425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:43.403627 14430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:43.403628 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:43.403710 14425 server_base.cc:1060] running on GCE node
W20260128 20:51:43.403728 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:43.404367 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:43.405004 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:43.406203 14425 hybrid_clock.cc:648] HybridClock initialized: now 1769633503406169 us; error 52 us; skew 500 ppm
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633503, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:43.409379 14425 init.cc:377] Logged in from keytab as oryx/127.4.183.130@KRBTEST.COM (short username oryx)
I20260128 20:51:43.410826 14425 webserver.cc:492] Webserver started at http://127.4.183.130:44405/ using document root <none> and password file <none>
I20260128 20:51:43.411530 14425 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:43.411638 14425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:43.416639 14425 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 20:51:43.419760 14440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.422214 14425 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260128 20:51:43.422462 14425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "22cf719fc0124ce9884e8fe52da7d612"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "e80a599474d7f5b5d1307d4c39713c7a"
server_key_iv: "ae08966da4721de7222832b86390e6f4"
server_key_version: "encryptionkey@0"
I20260128 20:51:43.423069 14425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:43.439632 14425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:43.443928 14425 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:43.444181 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:43.444824 14425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:43.445863 14425 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:43.446012 14425 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.446133 14425 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:43.446177 14425 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.458503 14425 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36309
I20260128 20:51:43.458564 14553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36309 every 8 connection(s)
I20260128 20:51:43.459728 14425 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:43.468552 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14425
I20260128 20:51:43.469112 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:45819
--local_ip_for_outbound_sockets=127.4.183.131
--tserver_master_addrs=127.4.183.190:39755
--webserver_port=33073
--webserver_interface=127.4.183.131
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.183.131
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 20:51:43.472426 14423 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.462040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:57911 (local address 127.4.183.190:39755)
0128 20:51:43.462194 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.462198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.463079 (+ 881us) server_negotiation.cc:408] Connection header received
0128 20:51:43.464107 (+ 1028us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.464111 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.464187 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.464300 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.466650 (+ 2350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.467186 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.467952 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.468140 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.472239 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:57911: BlockingRecv error: recv got EOF from 127.4.183.130:57911 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260128 20:51:43.472862 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.462293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:57911)
0128 20:51:43.462872 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.462920 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.463838 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.464517 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.464528 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.465024 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.466376 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.466398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.467342 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.467351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.467799 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.467808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.468123 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.468894 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.468927 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.471998 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":145,"threads_started":1}
W20260128 20:51:43.473568 14554 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:39755 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.183.190:39755: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.480930 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.474212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:51139)
0128 20:51:43.474386 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.474406 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.474509 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.475004 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.475007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.475209 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.475419 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.475426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.476254 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.476259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.476657 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.476666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.476813 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.477933 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.477948 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.480704 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260128 20:51:43.481101 14423 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.474533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:51139 (local address 127.4.183.190:39755)
0128 20:51:43.474744 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.474749 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.474764 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:43.474805 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.474807 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.474857 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.474971 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.475560 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.476115 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.476790 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.476984 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.480960 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:51139: BlockingRecv error: recv got EOF from 127.4.183.130:51139 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.488221 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.482199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:51617)
0128 20:51:43.482427 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.482449 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.482580 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.482943 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.482948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.483202 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.483490 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.483501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.484319 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.484323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.484972 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.484986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.485165 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.485757 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.485777 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.488059 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260128 20:51:43.488547 14423 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.482368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:51617 (local address 127.4.183.190:39755)
0128 20:51:43.482517 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.482520 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.482538 (+ 18us) server_negotiation.cc:408] Connection header received
0128 20:51:43.482743 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.482747 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.482805 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.482890 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.483644 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.484190 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.485128 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.485375 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.488244 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:51617: BlockingRecv error: recv got EOF from 127.4.183.130:51617 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260128 20:51:43.488550 14554 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260128 20:51:43.600274 14557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:43.600555 14557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:43.600656 14557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:43.604638 14557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:43.604737 14557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:43.604856 14557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:43.609808 14557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/krb5kdc/oryx_127.4.183.131.keytab
--principal=oryx/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:45819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=33073
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:43.611100 14557 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:43.612040 14557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:43.619263 14562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:43.619349 14557 server_base.cc:1060] running on GCE node
W20260128 20:51:43.619270 14565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:43.619249 14563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:43.620249 14557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:43.620966 14557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:43.622296 14557 hybrid_clock.cc:648] HybridClock initialized: now 1769633503622281 us; error 54 us; skew 500 ppm
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633503, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:51:43.625415 14557 init.cc:377] Logged in from keytab as oryx/127.4.183.131@KRBTEST.COM (short username oryx)
I20260128 20:51:43.626806 14557 webserver.cc:492] Webserver started at http://127.4.183.131:33073/ using document root <none> and password file <none>
I20260128 20:51:43.627457 14557 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:43.627557 14557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:43.632145 14557 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:51:43.634735 14572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.636126 14557 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:51:43.636319 14557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "457e0998265d4023ad3ff570a6c366fa"
format_stamp: "Formatted at 2026-01-28 20:51:42 on dist-test-slave-hss7"
server_key: "c4ca7ca9484bfe1e1bfd44395c33da2b"
server_key_iv: "433b84a6e10d0c09c038b678fd0ca691"
server_key_version: "encryptionkey@0"
I20260128 20:51:43.636829 14557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:43.658860 14557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:43.662086 14557 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:43.662310 14557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:43.662911 14557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:43.663862 14557 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:43.663934 14557 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.664016 14557 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:43.664055 14557 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:43.674638 14557 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:45819
I20260128 20:51:43.674652 14685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:45819 every 8 connection(s)
I20260128 20:51:43.675943 14557 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:43.677284 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14557
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.688196 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.678108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.4.183.190:39755)
0128 20:51:43.678285 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.678290 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.678612 (+ 322us) server_negotiation.cc:408] Connection header received
0128 20:51:43.678864 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.678869 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.678929 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.679038 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.680248 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.680753 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.681704 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.681985 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.683463 (+ 1478us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:43.683490 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:43.683508 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:51:43.683568 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:43.685799 (+ 2231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:43.686348 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:43.686356 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:43.686365 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:43.686488 (+ 123us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:51:43.686814 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:51:43.686816 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:51:43.686818 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:51:43.687242 (+ 424us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:43.687478 (+ 236us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:43.687843 (+ 365us) server_negotiation.cc:300] Negotiation successful
0128 20:51:43.688073 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260128 20:51:43.689014 14689 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.678422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:40647 (local address 127.4.183.190:39755)
0128 20:51:43.678683 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.678691 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.679259 (+ 568us) server_negotiation.cc:408] Connection header received
0128 20:51:43.680400 (+ 1141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.680404 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.680460 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.680551 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.683134 (+ 2583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.683900 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.684784 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.685018 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.688907 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:40647: BlockingRecv error: recv got EOF from 127.4.183.131:40647 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":101,"threads_started":1}
I20260128 20:51:43.689504 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.678478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:40647)
0128 20:51:43.679105 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.679156 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.680178 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.680742 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.680751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.681373 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.682921 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.682951 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.684051 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.684059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.684618 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.684636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.684886 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.685451 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.685480 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.688773 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":173,"threads_started":1}
W20260128 20:51:43.690053 14686 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:39755 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.183.190:39755: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.695655 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.690544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:48377)
0128 20:51:43.690700 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.690716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.690808 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.691069 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.691072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.691257 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.691445 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.691451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.692197 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.692201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.692588 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.692596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.692726 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.693390 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.693407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.695473 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260128 20:51:43.695793 14689 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.690582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:48377 (local address 127.4.183.190:39755)
0128 20:51:43.690700 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.690706 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.690753 (+ 47us) server_negotiation.cc:408] Connection header received
0128 20:51:43.690917 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.690925 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.690972 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.691050 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.691562 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.692079 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.692728 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.692939 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.695668 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:48377: BlockingRecv error: recv got EOF from 127.4.183.131:48377 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:43 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:43.701803 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:43.696328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:53621)
0128 20:51:43.696767 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:43.696784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:43.696951 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:43.697282 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:43.697285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:43.697437 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:43.697627 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.697633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.698426 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.698429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:43.698843 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:43.698852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.698962 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.699538 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:43.699553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:43.701629 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262}
W20260128 20:51:43.702172 14689 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:43.696502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:53621 (local address 127.4.183.190:39755)
0128 20:51:43.696702 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:43.696706 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:43.696924 (+ 218us) server_negotiation.cc:408] Connection header received
0128 20:51:43.697047 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:43.697050 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:43.697099 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:43.697259 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:43.697780 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.698294 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:43.699011 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:43.699157 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:43.701986 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:53621: BlockingRecv error: recv got EOF from 127.4.183.131:53621 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260128 20:51:43.702183 14686 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jan 28 20:51:44 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:44 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:44.288249 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:44.282298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:57269)
0128 20:51:44.282654 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:44.282669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:44.282806 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:44.283243 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:44.283247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:44.283495 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:44.283830 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:44.283843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.284694 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:44.284699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:44.285267 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:44.285279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.285401 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:44.286079 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:44.286094 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:44.288089 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
W20260128 20:51:44.288528 14692 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:44.282345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:57269 (local address 127.4.183.190:39755)
0128 20:51:44.282645 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:44.282648 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:44.282804 (+ 156us) server_negotiation.cc:408] Connection header received
0128 20:51:44.282928 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:44.282931 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:44.282990 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:44.283081 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:44.284014 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.284549 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:44.285412 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.285566 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:44.288384 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:57269: BlockingRecv error: recv got EOF from 127.4.183.129:57269 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":122,"threads_started":1}
Jan 28 20:51:44 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:44 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:44.498176 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:44.489415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:39991)
0128 20:51:44.489750 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:44.489766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:44.489877 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:44.490138 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:44.490141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:44.490363 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:44.490565 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:44.490571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.491539 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:44.491543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:44.492130 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:44.492140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.492271 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:44.495801 (+ 3530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:44.495827 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:44.498029 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
W20260128 20:51:44.498652 14692 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:44.489561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:39991 (local address 127.4.183.190:39755)
0128 20:51:44.489722 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:44.489726 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:44.489839 (+ 113us) server_negotiation.cc:408] Connection header received
0128 20:51:44.489963 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:44.489966 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:44.490018 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:44.490105 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:44.490730 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.491351 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:44.492251 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.492471 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:44.498288 (+ 5817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:39991: BlockingRecv error: recv got EOF from 127.4.183.130:39991 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jan 28 20:51:44 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:44 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:44.709178 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:44.703063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:38687)
0128 20:51:44.703386 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:44.703399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:44.703545 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:44.703842 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:44.703845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:44.704061 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:44.704274 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:44.704281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.705063 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:44.705069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:44.705561 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:44.705572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.705738 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:44.706659 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:44.706678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:44.708902 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
W20260128 20:51:44.709440 14692 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:44.703154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:38687 (local address 127.4.183.190:39755)
0128 20:51:44.703300 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:44.703304 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:44.703480 (+ 176us) server_negotiation.cc:408] Connection header received
0128 20:51:44.703671 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:44.703674 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:44.703726 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:44.703815 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:44.704414 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.704939 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:44.705716 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:44.705934 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:44.709185 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:38687: BlockingRecv error: recv got EOF from 127.4.183.131:38687 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 28 20:51:45 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:45 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:45.295624 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:45.289214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:41721)
0128 20:51:45.289508 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:45.289522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:45.289658 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:45.290134 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:45.290137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:45.290383 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:45.290640 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:45.290650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.291489 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:45.291493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:45.291912 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:45.291920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.292026 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:45.292782 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:45.292811 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:45.295445 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
W20260128 20:51:45.295770 14696 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:45.289266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:41721 (local address 127.4.183.190:39755)
0128 20:51:45.289528 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:45.289534 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:45.289578 (+ 44us) server_negotiation.cc:408] Connection header received
0128 20:51:45.289776 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:45.289779 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:45.289856 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:45.289982 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:45.290791 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.291347 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:45.292078 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.292295 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:45.295628 (+ 3333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:41721: BlockingRecv error: recv got EOF from 127.4.183.129:41721 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
Jan 28 20:51:45 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:45 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:45.505563 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:45.499441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:43143)
0128 20:51:45.499794 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:45.499815 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:45.499987 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:45.500289 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:45.500293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:45.500519 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:45.500773 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:45.500785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.501746 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:45.501750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:45.502149 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:45.502157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.502284 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:45.502959 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:45.502982 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:45.505429 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
W20260128 20:51:45.505815 14696 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:45.499623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:43143 (local address 127.4.183.190:39755)
0128 20:51:45.499798 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:45.499806 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:45.499929 (+ 123us) server_negotiation.cc:408] Connection header received
0128 20:51:45.500110 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:45.500114 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:45.500172 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:45.500263 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:45.500955 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.501625 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:45.502275 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.502483 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:45.505672 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:43143: BlockingRecv error: recv got EOF from 127.4.183.130:43143 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jan 28 20:51:45 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:45 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:45.715835 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:45.710137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:48485)
0128 20:51:45.710471 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:45.710485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:45.710608 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:45.710918 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:45.710920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:45.711104 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:45.711318 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:45.711324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.712128 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:45.712131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:45.712658 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:45.712670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.712797 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:45.713442 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:45.713459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:45.715692 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
W20260128 20:51:45.716029 14696 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:45.710358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:48485 (local address 127.4.183.190:39755)
0128 20:51:45.710498 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:45.710502 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:45.710542 (+ 40us) server_negotiation.cc:408] Connection header received
0128 20:51:45.710738 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:45.710741 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:45.710797 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:45.710876 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:45.711451 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.711991 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:45.712810 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:45.713012 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:45.715854 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:48485: BlockingRecv error: recv got EOF from 127.4.183.131:48485 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 28 20:51:46 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:46 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:46.301962 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:46.296594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:41175)
0128 20:51:46.296882 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:46.296896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:46.296984 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:46.297252 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:46.297254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:46.297440 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:46.297649 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:46.297655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.298516 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:46.298519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:46.298947 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:46.298955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.299061 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:46.299646 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:46.299666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:46.301815 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
W20260128 20:51:46.302181 14700 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:46.296719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:41175 (local address 127.4.183.190:39755)
0128 20:51:46.296972 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:46.296976 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:46.296991 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:46.297072 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:46.297075 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:46.297123 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:46.297226 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:46.297816 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.298376 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:46.299078 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.299229 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:46.302032 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:41175: BlockingRecv error: recv got EOF from 127.4.183.129:41175 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
Jan 28 20:51:46 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:46 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:46.513310 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:46.506792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:40463)
0128 20:51:46.507155 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:46.507177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:46.507327 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:46.507771 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:46.507774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:46.508016 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:46.508283 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:46.508291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.509617 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:46.509626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:46.510133 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:46.510146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.510327 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:46.511010 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:46.511031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:46.513116 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
W20260128 20:51:46.513653 14700 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:46.506977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:40463 (local address 127.4.183.190:39755)
0128 20:51:46.507197 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:46.507201 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:46.507277 (+ 76us) server_negotiation.cc:408] Connection header received
0128 20:51:46.507471 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:46.507475 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:46.507552 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:46.507673 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:46.508617 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.509373 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:46.510464 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.510922 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:46.513410 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:40463: BlockingRecv error: recv got EOF from 127.4.183.130:40463 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jan 28 20:51:46 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:46 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:46.722826 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:46.716905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:49029)
0128 20:51:46.717271 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:46.717295 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:46.717647 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:46.718045 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:46.718048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:46.718263 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:46.718460 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:46.718467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.719238 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:46.719242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:46.719628 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:46.719636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.719784 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:46.720447 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:46.720463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:46.722677 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":103,"threads_started":1}
W20260128 20:51:46.723023 14700 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:46.717004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:49029 (local address 127.4.183.190:39755)
0128 20:51:46.717235 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:46.717241 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:46.717379 (+ 138us) server_negotiation.cc:408] Connection header received
0128 20:51:46.717849 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:46.717853 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:46.717917 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:46.718017 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:46.718596 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.719112 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:46.719760 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:46.719966 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:46.722875 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:49029: BlockingRecv error: recv got EOF from 127.4.183.131:49029 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jan 28 20:51:47 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:47 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:47.309047 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:47.302801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:60367)
0128 20:51:47.303122 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:47.303136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:47.303228 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:47.303587 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:47.303589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:47.303822 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:47.304027 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:47.304033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.304921 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:47.304925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:47.305427 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:47.305437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.305545 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:47.306381 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:47.306404 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:47.308843 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":131,"threads_started":1}
W20260128 20:51:47.309193 14704 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:47.302936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:60367 (local address 127.4.183.190:39755)
0128 20:51:47.303180 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:47.303183 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:47.303198 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:47.303370 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:47.303381 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:47.303433 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:47.303526 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:47.304177 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.304746 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:47.305573 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.305736 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:47.309025 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:60367: BlockingRecv error: recv got EOF from 127.4.183.129:60367 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
Jan 28 20:51:47 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:47 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:47.520812 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:47.514127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:41091)
0128 20:51:47.514538 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:47.514561 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:47.514724 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:47.515046 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:47.515050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:47.515265 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:47.515464 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:47.515470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.516256 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:47.516261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:47.516657 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:47.516665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.516827 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:47.517696 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:47.517711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:47.520267 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":118,"threads_started":1}
W20260128 20:51:47.521005 14704 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:47.514335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:41091 (local address 127.4.183.190:39755)
0128 20:51:47.514525 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:47.514530 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:47.514658 (+ 128us) server_negotiation.cc:408] Connection header received
0128 20:51:47.514852 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:47.514856 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:47.514926 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:47.515026 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:47.515602 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.516128 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:47.516802 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.517025 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:47.520837 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:41091: BlockingRecv error: recv got EOF from 127.4.183.130:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jan 28 20:51:47 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:47 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:47.730015 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:47.724285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:56681)
0128 20:51:47.724620 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:47.724639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:47.724791 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:47.725082 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:47.725084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:47.725315 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:47.725517 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:47.725524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.726328 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:47.726335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:47.726741 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:47.726749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.726881 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:47.727477 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:47.727492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:47.729881 (+ 2389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
W20260128 20:51:47.730350 14704 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:47.724444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:56681 (local address 127.4.183.190:39755)
0128 20:51:47.724628 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:47.724633 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:47.724732 (+ 99us) server_negotiation.cc:408] Connection header received
0128 20:51:47.724885 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:47.724888 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:47.724942 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:47.725034 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:47.725664 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.726191 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:47.726873 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:47.727102 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:47.730230 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:56681: BlockingRecv error: recv got EOF from 127.4.183.131:56681 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jan 28 20:51:48 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:48 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:48.316771 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:48.310124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:48849)
0128 20:51:48.310498 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:48.310514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:48.310618 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:48.311053 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:48.311056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:48.311254 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:48.311486 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:48.311492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.312490 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:48.312497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:48.313034 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:48.313044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.313236 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:48.313877 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:48.313892 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:48.316622 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
W20260128 20:51:48.317075 14708 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:48.310303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:48849 (local address 127.4.183.190:39755)
0128 20:51:48.310553 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:48.310556 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:48.310573 (+ 17us) server_negotiation.cc:408] Connection header received
0128 20:51:48.310782 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:48.310786 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:48.310839 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:48.310931 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:48.311654 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.312368 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:48.313211 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.313383 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:48.316935 (+ 3552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:48849: BlockingRecv error: recv got EOF from 127.4.183.129:48849 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
Jan 28 20:51:48 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:48 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:48.527463 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:48.521966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:39803)
0128 20:51:48.522301 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:48.522315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:48.522414 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:48.522699 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:48.522701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:48.522892 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:48.523107 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:48.523114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.523904 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:48.523906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:48.524291 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:48.524298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.524390 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:48.524992 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:48.525006 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:48.527298 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":139,"threads_started":1}
W20260128 20:51:48.527726 14708 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:48.522179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:39803 (local address 127.4.183.190:39755)
0128 20:51:48.522329 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:48.522333 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:48.522369 (+ 36us) server_negotiation.cc:408] Connection header received
0128 20:51:48.522504 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:48.522507 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:48.522554 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:48.522639 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:48.523256 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.523801 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:48.524390 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.524525 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:48.527592 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:39803: BlockingRecv error: recv got EOF from 127.4.183.130:39803 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jan 28 20:51:48 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:48 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:48.736454 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:48.731018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:50703)
0128 20:51:48.731349 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:48.731364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:48.731482 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:48.731814 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:48.731817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:48.732011 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:48.732205 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:48.732211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.733040 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:48.733043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:48.733535 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:48.733544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.733650 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:48.734259 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:48.734273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:48.736280 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
W20260128 20:51:48.736552 14708 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:48.731193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:50703 (local address 127.4.183.190:39755)
0128 20:51:48.731359 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:48.731363 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:48.731444 (+ 81us) server_negotiation.cc:408] Connection header received
0128 20:51:48.731622 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:48.731625 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:48.731678 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:48.731763 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:48.732314 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.732812 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:48.733674 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:48.733822 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:48.736425 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:50703: BlockingRecv error: recv got EOF from 127.4.183.131:50703 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jan 28 20:51:49 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:49 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:49.325596 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:49.318170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:44809)
0128 20:51:49.318668 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:49.318687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:49.318833 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:49.319220 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:49.319223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:49.319489 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:49.319765 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:49.319772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.320867 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:49.320871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:49.321460 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:49.321473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.321713 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:49.322687 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:49.322707 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:49.325454 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":189,"threads_started":1}
W20260128 20:51:49.325870 14712 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:49.318382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:44809 (local address 127.4.183.190:39755)
0128 20:51:49.318863 (+ 481us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:49.318868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:49.318925 (+ 57us) server_negotiation.cc:408] Connection header received
0128 20:51:49.318973 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:49.318975 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:49.319028 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:49.319115 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:49.320019 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.320592 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:49.321684 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.321883 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:49.325712 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:44809: BlockingRecv error: recv got EOF from 127.4.183.129:44809 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":78,"threads_started":1}
Jan 28 20:51:49 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:49 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:49.538759 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:49.528452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:57331)
0128 20:51:49.528744 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:49.528758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:49.528865 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:49.529433 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:49.529436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:49.529634 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:49.529857 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:49.529863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.531044 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:49.531051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:49.531473 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:49.531482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.531617 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:49.536199 (+ 4582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:49.536219 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:49.538632 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
W20260128 20:51:49.539022 14712 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:49.528549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:57331 (local address 127.4.183.190:39755)
0128 20:51:49.528754 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:49.528758 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:49.528834 (+ 76us) server_negotiation.cc:408] Connection header received
0128 20:51:49.529091 (+ 257us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:49.529096 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:49.529228 (+ 132us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:49.529348 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:49.530056 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.530860 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:49.531664 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.531901 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:49.538858 (+ 6957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:57331: BlockingRecv error: recv got EOF from 127.4.183.130:57331 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jan 28 20:51:49 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:49 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:49.744689 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:49.737961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:49195)
0128 20:51:49.738403 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:49.738423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:49.738599 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:49.739025 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:49.739029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:49.739230 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:49.739436 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:49.739443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.740397 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:49.740400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:49.740821 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:49.740830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.741009 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:49.741802 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:49.741867 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:49.744512 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":92,"threads_started":1}
W20260128 20:51:49.744812 14712 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:49.738213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:49195 (local address 127.4.183.190:39755)
0128 20:51:49.738433 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:49.738437 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:49.738543 (+ 106us) server_negotiation.cc:408] Connection header received
0128 20:51:49.738712 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:49.738716 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:49.738781 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:49.738886 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:49.739610 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.740227 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:49.740997 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:49.741253 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:49.744689 (+ 3436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:49195: BlockingRecv error: recv got EOF from 127.4.183.131:49195 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jan 28 20:51:50 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:50 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:50.333633 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:50.326635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:44379)
0128 20:51:50.327001 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:50.327019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:50.327128 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:50.327543 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:50.327548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:50.327925 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:50.328218 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:50.328229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.329536 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:50.329544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:50.330118 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:50.330130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.330309 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:50.330998 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:50.331014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:50.333442 (+ 2428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":87,"threads_started":1}
W20260128 20:51:50.333926 14716 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:50.326805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:44379 (local address 127.4.183.190:39755)
0128 20:51:50.327113 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:50.327116 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:50.327133 (+ 17us) server_negotiation.cc:408] Connection header received
0128 20:51:50.327207 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:50.327210 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:50.327263 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:50.327355 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:50.328474 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.329284 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:50.330318 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.330559 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:50.333712 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:44379: BlockingRecv error: recv got EOF from 127.4.183.129:44379 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
Jan 28 20:51:50 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:50 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:50.546725 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:50.539776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:43103)
0128 20:51:50.540202 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:50.540215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:50.540364 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:50.541072 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:50.541076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:50.541425 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:50.541807 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:50.541819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.542874 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:50.542878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:50.543319 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:50.543328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.543445 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:50.544105 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:50.544124 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:50.546595 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":123,"threads_started":1}
W20260128 20:51:50.546983 14716 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:50.539830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:43103 (local address 127.4.183.190:39755)
0128 20:51:50.540043 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:50.540047 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:50.540327 (+ 280us) server_negotiation.cc:408] Connection header received
0128 20:51:50.540675 (+ 348us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:50.540681 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:50.540790 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:50.540907 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:50.542117 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.542690 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:50.543485 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.543652 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:50.546850 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:43103: BlockingRecv error: recv got EOF from 127.4.183.130:43103 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jan 28 20:51:50 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:50 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:50.751483 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:50.745793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:43729)
0128 20:51:50.746186 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:50.746211 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:50.746373 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:50.746685 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:50.746689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:50.746937 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:50.747143 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:50.747151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.747961 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:50.747967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:50.748363 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:50.748372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.748524 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:50.749215 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:50.749235 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:50.751344 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
W20260128 20:51:50.751763 14716 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:50.745853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:43729 (local address 127.4.183.190:39755)
0128 20:51:50.746136 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:50.746142 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:50.746303 (+ 161us) server_negotiation.cc:408] Connection header received
0128 20:51:50.746491 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:50.746495 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:50.746560 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:50.746671 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:50.747295 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.747819 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:50.748499 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:50.748711 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:50.751574 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:43729: BlockingRecv error: recv got EOF from 127.4.183.131:43729 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Jan 28 20:51:51 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:51 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:51.341598 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:51.334729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:37665)
0128 20:51:51.335183 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:51.335204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:51.335325 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:51.335964 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:51.335967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:51.336166 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:51.336377 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:51.336384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.337516 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:51.337519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:51.338108 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:51.338118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.338227 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:51.338921 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:51.338944 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:51.341406 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":279,"threads_started":1}
W20260128 20:51:51.341744 14720 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:51.335026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:37665 (local address 127.4.183.190:39755)
0128 20:51:51.335287 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:51.335291 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:51.335307 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:51:51.335686 (+ 379us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:51.335694 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:51.335789 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:51.335895 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:51.336570 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.337371 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:51.338258 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.338412 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:51.341598 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:37665: BlockingRecv error: recv got EOF from 127.4.183.129:37665 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
Jan 28 20:51:51 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:51 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:51.554072 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:51.547737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:57621)
0128 20:51:51.548050 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:51.548081 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:51.548193 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:51.548527 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:51.548530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:51.548739 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:51.548944 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:51.548951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.550239 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:51.550243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:51.550665 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:51.550674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.550838 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:51.551504 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:51.551518 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:51.553894 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
W20260128 20:51:51.554358 14720 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:51.547939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:57621 (local address 127.4.183.190:39755)
0128 20:51:51.548102 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:51.548106 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:51.548143 (+ 37us) server_negotiation.cc:408] Connection header received
0128 20:51:51.548328 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:51.548331 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:51.548391 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:51.548481 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:51.549227 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.550042 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:51.550894 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.551093 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:51.554164 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:57621: BlockingRecv error: recv got EOF from 127.4.183.130:57621 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jan 28 20:51:51 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:51 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:51.758740 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:51.752591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:50855)
0128 20:51:51.752943 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:51.752960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:51.753136 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:51.753512 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:51.753515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:51.753756 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:51.753979 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:51.753986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.754866 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:51.754870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:51.755278 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:51.755287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.755400 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:51.756038 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:51.756052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:51.758485 (+ 2433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
W20260128 20:51:51.758872 14720 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:51.752757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:50855 (local address 127.4.183.190:39755)
0128 20:51:51.752968 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:51.752972 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:51.753093 (+ 121us) server_negotiation.cc:408] Connection header received
0128 20:51:51.753296 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:51.753300 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:51.753359 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:51.753443 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:51.754200 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.754720 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:51.755414 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:51.755557 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:51.758730 (+ 3173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:50855: BlockingRecv error: recv got EOF from 127.4.183.131:50855 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jan 28 20:51:52 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:52 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:52.348788 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:52.342742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:57173)
0128 20:51:52.343111 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:52.343127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:52.343237 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:52.343628 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:52.343630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:52.343839 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:52.344210 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:52.344230 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.345343 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:52.345347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:52.345791 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:52.345802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.345936 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:52.346541 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:52.346555 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:52.348641 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
W20260128 20:51:52.349227 14724 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:52.342896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:57173 (local address 127.4.183.190:39755)
0128 20:51:52.343291 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:52.343296 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:52.343315 (+ 19us) server_negotiation.cc:408] Connection header received
0128 20:51:52.343375 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:52.343378 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:52.343442 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:52.343579 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:52.344501 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.345083 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:52.345938 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.346097 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:52.349009 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:57173: BlockingRecv error: recv got EOF from 127.4.183.129:57173 (error 108)
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":167,"threads_started":1}
Jan 28 20:51:52 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:52 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:52.560547 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:52.555168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:54183)
0128 20:51:52.555478 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:52.555493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:52.555602 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:52.556019 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:52.556022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:52.556224 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:52.556428 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:52.556434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.557329 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:52.557333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:52.557736 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:52.557783 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.557904 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:52.558547 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:52.558560 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:52.560419 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
W20260128 20:51:52.560786 14724 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:52.555345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:54183 (local address 127.4.183.190:39755)
0128 20:51:52.555533 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:52.555537 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:52.555552 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:52.555804 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:52.555807 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:52.555866 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:52.555948 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:52.556590 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.557122 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:52.557959 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.558117 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:52.560628 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:54183: BlockingRecv error: recv got EOF from 127.4.183.130:54183 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jan 28 20:51:52 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:52 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:52.765558 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:52.760034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:42477)
0128 20:51:52.760497 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:52.760511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:52.760635 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:52.760999 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:52.761003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:52.761288 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:52.761493 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:52.761500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.762240 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:52.762244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:52.762654 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:52.762669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.762807 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:52.763415 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:52.763429 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:52.765430 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":88,"threads_started":1}
W20260128 20:51:52.765807 14724 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:52.760293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:42477 (local address 127.4.183.190:39755)
0128 20:51:52.760513 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:52.760520 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:52.760602 (+ 82us) server_negotiation.cc:408] Connection header received
0128 20:51:52.760786 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:52.760789 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:52.760844 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:52.760949 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:52.761616 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.762120 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:52.762834 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:52.762979 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:52.765679 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:42477: BlockingRecv error: recv got EOF from 127.4.183.131:42477 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jan 28 20:51:53 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:53 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:53.355365 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:53.349822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:40969)
0128 20:51:53.350203 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:53.350225 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:53.350344 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:53.350647 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:53.350650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:53.350877 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:53.351089 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:53.351096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.351880 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:53.351883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:53.352296 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:53.352303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.352474 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:53.353234 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:53.353252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:53.355238 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
W20260128 20:51:53.355546 14728 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:53.350004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:40969 (local address 127.4.183.190:39755)
0128 20:51:53.350310 (+ 306us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:53.350314 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:53.350330 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:51:53.350444 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:53.350447 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:53.350494 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:53.350579 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:53.351223 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.351752 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:53.352451 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.352660 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:53.355403 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:40969: BlockingRecv error: recv got EOF from 127.4.183.129:40969 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
Jan 28 20:51:53 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:53 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:53.568462 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:53.561677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:37989)
0128 20:51:53.562403 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:53.562429 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:53.562563 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:53.563087 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:53.563089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:53.563288 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:53.563503 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:53.563510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.564457 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:53.564460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:53.564876 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:53.564884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.565009 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:53.566021 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:53.566046 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:53.568336 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":546,"threads_started":1}
W20260128 20:51:53.568619 14728 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:53.561945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:37989 (local address 127.4.183.190:39755)
0128 20:51:53.562152 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:53.562156 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:53.562714 (+ 558us) server_negotiation.cc:408] Connection header received
0128 20:51:53.562811 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:53.562815 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:53.562903 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:53.563029 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:53.563769 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.564304 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:53.565048 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.565321 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:53.568496 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:37989: BlockingRecv error: recv got EOF from 127.4.183.130:37989 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jan 28 20:51:53 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:53 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:53.772797 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:53.766718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:60035)
0128 20:51:53.767167 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:53.767187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:53.767358 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:53.767735 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:53.767738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:53.768040 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:53.768277 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:53.768286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.769084 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:53.769090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:53.769579 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:53.769589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.769755 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:53.770416 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:53.770430 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:53.772508 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
W20260128 20:51:53.772912 14728 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:53.767027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:60035 (local address 127.4.183.190:39755)
0128 20:51:53.767248 (+ 221us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:53.767256 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:53.767275 (+ 19us) server_negotiation.cc:408] Connection header received
0128 20:51:53.767525 (+ 250us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:53.767528 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:53.767590 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:53.767698 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:53.768422 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.768950 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:53.769739 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:53.769957 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:53.772770 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:60035: BlockingRecv error: recv got EOF from 127.4.183.131:60035 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jan 28 20:51:54 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:54 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:54.366767 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:54.356449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:59933)
0128 20:51:54.356825 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:54.356845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:54.356965 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:54.357525 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:54.357528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:54.357734 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:54.357954 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:54.357961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.359117 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:54.359126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:54.359730 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:54.359748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.359862 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:54.364511 (+ 4649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:54.364532 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:54.366630 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":82,"threads_started":1}
W20260128 20:51:54.367079 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:54.356639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:59933 (local address 127.4.183.190:39755)
0128 20:51:54.357119 (+ 480us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:54.357124 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:54.357200 (+ 76us) server_negotiation.cc:408] Connection header received
0128 20:51:54.357261 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:54.357264 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:54.357339 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:54.357466 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:54.358163 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.358924 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:54.359875 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.360045 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:54.366933 (+ 6888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:59933: BlockingRecv error: recv got EOF from 127.4.183.129:59933 (error 108)
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":90,"threads_started":1}
Jan 28 20:51:54 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:54 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:54.578936 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:54.569482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:36047)
0128 20:51:54.569794 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:54.569835 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:54.569988 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:54.570431 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:54.570435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:54.570673 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:54.570913 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:54.570927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.571838 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:54.571842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:54.572257 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:54.572267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.572380 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:54.576379 (+ 3999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:54.576399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:54.578801 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
W20260128 20:51:54.579260 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:54.569633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:36047 (local address 127.4.183.190:39755)
0128 20:51:54.569859 (+ 226us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:54.569866 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:54.569917 (+ 51us) server_negotiation.cc:408] Connection header received
0128 20:51:54.570116 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:54.570119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:54.570199 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:54.570307 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:54.571082 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.571686 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:54.572415 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.572576 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:54.579157 (+ 6581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:36047: BlockingRecv error: recv got EOF from 127.4.183.130:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jan 28 20:51:54 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:54 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:54.779844 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:54.773827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:60401)
0128 20:51:54.774180 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:54.774198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:54.774376 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:54.774677 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:54.774680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:54.774892 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:54.775088 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:54.775094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.775864 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:54.775869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:54.776266 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:54.776273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.776409 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:54.777052 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:54.777068 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:54.779721 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
W20260128 20:51:54.780119 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:54.774025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:60401 (local address 127.4.183.190:39755)
0128 20:51:54.774182 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:54.774189 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:54.774290 (+ 101us) server_negotiation.cc:408] Connection header received
0128 20:51:54.774487 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:54.774493 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:54.774550 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:54.774659 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:54.775222 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.775743 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:54.776389 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:54.776580 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:54.779893 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:60401: BlockingRecv error: recv got EOF from 127.4.183.131:60401 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jan 28 20:51:55 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:55 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:55.374125 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:55.368021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:44739)
0128 20:51:55.368393 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:55.368418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:55.368514 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:55.369099 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:55.369105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:55.369358 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:55.369672 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:55.369681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.370734 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:55.370738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:55.371159 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:55.371166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.371274 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:55.371870 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:55.371884 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:55.373973 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
W20260128 20:51:55.374814 14736 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:55.368289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:44739 (local address 127.4.183.190:39755)
0128 20:51:55.368547 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:55.368551 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:55.368565 (+ 14us) server_negotiation.cc:408] Connection header received
0128 20:51:55.368604 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:55.368609 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:55.368885 (+ 276us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:55.369030 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:55.369845 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.370578 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:55.371316 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.371472 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:55.374457 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:44739: BlockingRecv error: recv got EOF from 127.4.183.129:44739 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
Jan 28 20:51:55 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:55 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:55.586076 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:55.580367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:47501)
0128 20:51:55.580839 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:55.580854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:55.580970 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:55.581367 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:55.581370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:55.581571 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:55.581814 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:55.581821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.582753 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:55.582756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:55.583151 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:55.583158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.583263 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:55.583885 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:55.583899 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:55.585944 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":80,"threads_started":1}
W20260128 20:51:55.586392 14736 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:55.580551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:47501 (local address 127.4.183.190:39755)
0128 20:51:55.580890 (+ 339us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:55.580895 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:55.580917 (+ 22us) server_negotiation.cc:408] Connection header received
0128 20:51:55.581069 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:55.581071 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:55.581166 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:55.581274 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:55.581962 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.582573 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:55.583295 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.583451 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:55.586169 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:47501: BlockingRecv error: recv got EOF from 127.4.183.130:47501 (error 108)
Metrics: {"server-negotiator.queue_time_us":208}
Jan 28 20:51:55 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:55 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:55.787010 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:55.780804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:52965)
0128 20:51:55.781109 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:55.781124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:55.781291 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:55.781576 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:55.781580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:55.781774 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:55.781970 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:55.781976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.782710 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:55.782712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:55.783115 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:55.783123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.783218 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:55.784149 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:55.784171 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:55.786864 (+ 2693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
W20260128 20:51:55.787264 14736 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:55.780850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:52965 (local address 127.4.183.190:39755)
0128 20:51:55.780998 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:55.781001 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:55.781250 (+ 249us) server_negotiation.cc:408] Connection header received
0128 20:51:55.781368 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:55.781370 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:55.781422 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:55.781497 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:55.782097 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.782612 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:55.783248 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:55.783405 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:55.787131 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:52965: BlockingRecv error: recv got EOF from 127.4.183.131:52965 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jan 28 20:51:56 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:56 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:56.382669 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:56.375503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:40541)
0128 20:51:56.375914 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:56.375930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:56.376062 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:56.376433 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:56.376436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:56.376959 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:56.377295 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:56.377312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.378412 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:56.378418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:56.379118 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:56.379134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.379332 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:56.380013 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:56.380034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:56.382487 (+ 2453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":107,"threads_started":1}
W20260128 20:51:56.382936 14740 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:56.375665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:40541 (local address 127.4.183.190:39755)
0128 20:51:56.376022 (+ 357us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:56.376029 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:56.376058 (+ 29us) server_negotiation.cc:408] Connection header received
0128 20:51:56.376226 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:56.376231 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:56.376295 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:56.376412 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:56.377470 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.378047 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:56.379285 (+ 1238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.379540 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:56.382780 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:40541: BlockingRecv error: recv got EOF from 127.4.183.129:40541 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":154,"threads_started":1}
Jan 28 20:51:56 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:56 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:56.592556 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:56.587052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:54263)
0128 20:51:56.587434 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:56.587453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:56.587623 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:56.587924 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:56.587927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:56.588155 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:56.588347 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:56.588354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.589175 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:56.589179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:56.589554 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:56.589562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.589700 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:56.590359 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:56.590377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:56.592388 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
W20260128 20:51:56.592641 14740 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:56.587263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:54263 (local address 127.4.183.190:39755)
0128 20:51:56.587437 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:56.587443 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:56.587550 (+ 107us) server_negotiation.cc:408] Connection header received
0128 20:51:56.587755 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:56.587758 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:56.587810 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:56.587904 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:56.588494 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.589015 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:56.589688 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.589903 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:56.592527 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:54263: BlockingRecv error: recv got EOF from 127.4.183.130:54263 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jan 28 20:51:56 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:56 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:56.793668 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:56.788058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:55229)
0128 20:51:56.788359 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:56.788376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:56.788549 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:56.788893 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:56.788901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:56.789124 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:56.789385 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:56.789393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.790161 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:56.790165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:56.790560 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:56.790571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.790707 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:56.791312 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:56.791325 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:56.793537 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
W20260128 20:51:56.794072 14740 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:56.788177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:55229 (local address 127.4.183.190:39755)
0128 20:51:56.788359 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:56.788365 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:56.788482 (+ 117us) server_negotiation.cc:408] Connection header received
0128 20:51:56.788664 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:56.788669 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:56.788766 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:56.788865 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:56.789514 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.790031 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:56.790691 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:56.790898 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:56.793951 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:55229: BlockingRecv error: recv got EOF from 127.4.183.131:55229 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jan 28 20:51:57 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:57 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:57.389775 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:57.383730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:50835)
0128 20:51:57.384067 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:57.384083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:57.384181 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:57.384633 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:57.384636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:57.384854 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:57.385061 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:57.385069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.386063 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:57.386067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:57.386464 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:57.386472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.386582 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:57.387225 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:57.387240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:57.389607 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
W20260128 20:51:57.390055 14744 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:57.384098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:50835 (local address 127.4.183.190:39755)
0128 20:51:57.384373 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:57.384376 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:57.384391 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:51:57.384431 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:57.384434 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:57.384481 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:57.384580 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:57.385291 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.385883 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:57.386600 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.386797 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:57.389851 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:50835: BlockingRecv error: recv got EOF from 127.4.183.129:50835 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
Jan 28 20:51:57 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:57 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:57.598766 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:57.593362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:49383)
0128 20:51:57.593630 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:57.593648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:57.593822 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:57.594116 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:57.594119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:57.594333 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:57.594521 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:57.594527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.595284 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:57.595289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:57.595668 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:57.595674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.595810 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:57.596409 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:57.596422 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:57.598633 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
W20260128 20:51:57.599320 14744 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:57.593493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:49383 (local address 127.4.183.190:39755)
0128 20:51:57.593630 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:57.593636 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:57.593738 (+ 102us) server_negotiation.cc:408] Connection header received
0128 20:51:57.593946 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:57.593949 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:57.594002 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:57.594098 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:57.594654 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.595170 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:57.595784 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.595986 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:57.599222 (+ 3236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:49383: BlockingRecv error: recv got EOF from 127.4.183.130:49383 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jan 28 20:51:57 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:57 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.131@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:57.800107 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:57.794769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.131:41863)
0128 20:51:57.795060 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:57.795074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:57.795181 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:57.795440 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:57.795442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:57.795621 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:57.795847 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:57.795854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.796609 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:57.796611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:57.797008 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:57.797015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.797108 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:57.797741 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:57.797755 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:57.799916 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
W20260128 20:51:57.800143 14744 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:57.794900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:41863 (local address 127.4.183.190:39755)
0128 20:51:57.795062 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:57.795066 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:57.795158 (+ 92us) server_negotiation.cc:408] Connection header received
0128 20:51:57.795267 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:57.795270 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:57.795320 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:57.795406 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:57.796004 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.796504 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:57.797105 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:57.797326 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:57.800044 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.131:41863: BlockingRecv error: recv got EOF from 127.4.183.131:41863 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jan 28 20:51:58 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:58 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.129@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:58.396927 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:58.391073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.129:56263)
0128 20:51:58.391373 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:58.391388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:58.391481 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:58.391793 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:58.391798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:58.392043 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:58.392258 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:58.392265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.393114 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:58.393120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:58.393673 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:58.393685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.393822 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:58.394460 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:58.394474 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:58.396785 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":123,"threads_started":1}
W20260128 20:51:58.397367 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:58.391252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:56263 (local address 127.4.183.190:39755)
0128 20:51:58.391495 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:58.391498 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:58.391515 (+ 17us) server_negotiation.cc:408] Connection header received
0128 20:51:58.391560 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:58.391563 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:58.391608 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:58.391726 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:58.392422 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.392974 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:58.393819 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.394032 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:58.397016 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.129:56263: BlockingRecv error: recv got EOF from 127.4.183.129:56263 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
Jan 28 20:51:58 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 20:51:58 dist-test-slave-hss7 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.183.130@KRBTEST.COM for oryx/127.4.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 20:51:58.606477 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:58.599886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:39755 (local address 127.4.183.130:38063)
0128 20:51:58.600175 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:58.600198 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:58.600297 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:58.600926 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:58.600929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:58.601212 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:58.601448 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:58.601462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.602698 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:58.602703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:58.603187 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:58.603198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.603412 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:58.604179 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:51:58.604195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:58.606339 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.183.190:39755: Server oryx/127.4.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
W20260128 20:51:58.606817 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:58.600097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:38063 (local address 127.4.183.190:39755)
0128 20:51:58.600249 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:58.600252 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:58.600269 (+ 17us) server_negotiation.cc:408] Connection header received
0128 20:51:58.600595 (+ 326us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:58.600601 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:58.600703 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:58.600838 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:58.601672 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.602491 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:58.603342 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.603517 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:58.606664 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.183.130:38063: BlockingRecv error: recv got EOF from 127.4.183.130:38063 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260128 20:51:58.678009 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14273
I20260128 20:51:58.684979 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14425
I20260128 20:51:58.691337 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14557
I20260128 20:51:58.697628 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14203
2026-01-28T20:51:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19282 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-01-28T20:51:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:58Z Disabled control of system clock
I20260128 20:51:58.730299 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:36161
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260128 20:51:58.852653 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:58.852967 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:58.853019 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:58.857040 14753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:58.857156 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:58.857175 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:58.857195 14753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:58.857213 14753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:58.862293 14753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:36161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:58.863909 14753 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:58.865249 14753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:58.872440 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:58.872442 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:58.872442 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:58.872555 14753 server_base.cc:1060] running on GCE node
I20260128 20:51:58.873672 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:58.874900 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:58.876163 14753 hybrid_clock.cc:648] HybridClock initialized: now 1769633518876133 us; error 59 us; skew 500 ppm
I20260128 20:51:58.878937 14753 webserver.cc:492] Webserver started at http://127.4.183.190:39317/ using document root <none> and password file <none>
I20260128 20:51:58.879909 14753 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:58.880056 14753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:58.880362 14753 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:58.882619 14753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "5379a13e500442bb987390be91fb6364"
format_stamp: "Formatted at 2026-01-28 20:51:58 on dist-test-slave-hss7"
server_key: "eef4a23a0c0e0604cf9facee56a890b5"
server_key_iv: "105c48c8bea45c6692109b9e9b701aec"
server_key_version: "encryptionkey@0"
I20260128 20:51:58.883281 14753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "5379a13e500442bb987390be91fb6364"
format_stamp: "Formatted at 2026-01-28 20:51:58 on dist-test-slave-hss7"
server_key: "eef4a23a0c0e0604cf9facee56a890b5"
server_key_iv: "105c48c8bea45c6692109b9e9b701aec"
server_key_version: "encryptionkey@0"
I20260128 20:51:58.887594 14753 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:51:58.891181 14767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:58.893646 14753 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260128 20:51:58.893966 14753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "5379a13e500442bb987390be91fb6364"
format_stamp: "Formatted at 2026-01-28 20:51:58 on dist-test-slave-hss7"
server_key: "eef4a23a0c0e0604cf9facee56a890b5"
server_key_iv: "105c48c8bea45c6692109b9e9b701aec"
server_key_version: "encryptionkey@0"
I20260128 20:51:58.894147 14753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:58.947037 14753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:58.948125 14753 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:58.948352 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:58.956832 14753 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:36161
I20260128 20:51:58.956836 14819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:36161 every 8 connection(s)
I20260128 20:51:58.958024 14753 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:58.958328 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14753
I20260128 20:51:58.958462 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:58.958770 4830 external_mini_cluster.cc:1468] Setting key c4de881026242c2ee5b586c47c82ba9f
I20260128 20:51:58.962291 14820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:58.968421 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364: Bootstrap starting.
I20260128 20:51:58.968652 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:58.960419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.4.183.190:36161)
0128 20:51:58.961089 (+ 670us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:58.961099 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:58.961200 (+ 101us) server_negotiation.cc:408] Connection header received
0128 20:51:58.961931 (+ 731us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:58.961951 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:58.962350 (+ 399us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:58.962913 (+ 563us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:58.963554 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.964814 (+ 1260us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:58.965728 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:58.966091 (+ 363us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:58.966686 (+ 595us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:58.966744 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:58.966760 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:58.967236 (+ 476us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:58.967290 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:58.967308 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:58.967469 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:58.967656 (+ 187us) server_negotiation.cc:300] Negotiation successful
0128 20:51:58.967937 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":147,"threads_started":1}
I20260128 20:51:58.971261 14820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:58.972051 14820 log.cc:826] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:58.974315 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364: No bootstrap required, opened a new log
I20260128 20:51:58.977247 14820 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5379a13e500442bb987390be91fb6364" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 36161 } }
I20260128 20:51:58.977531 14820 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:58.977645 14820 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5379a13e500442bb987390be91fb6364, State: Initialized, Role: FOLLOWER
I20260128 20:51:58.978158 14820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5379a13e500442bb987390be91fb6364" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 36161 } }
I20260128 20:51:58.978295 14820 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:58.978339 14820 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:58.978415 14820 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:58.979382 14820 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5379a13e500442bb987390be91fb6364" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 36161 } }
I20260128 20:51:58.979737 14820 leader_election.cc:304] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5379a13e500442bb987390be91fb6364; no voters:
I20260128 20:51:58.980031 14820 leader_election.cc:290] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:58.980202 14825 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:58.980479 14825 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [term 1 LEADER]: Becoming Leader. State: Replica: 5379a13e500442bb987390be91fb6364, State: Running, Role: LEADER
I20260128 20:51:58.980805 14825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5379a13e500442bb987390be91fb6364" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 36161 } }
I20260128 20:51:58.981436 14820 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:58.982683 14826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5379a13e500442bb987390be91fb6364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5379a13e500442bb987390be91fb6364" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 36161 } } }
I20260128 20:51:58.982843 14826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:58.982729 14827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5379a13e500442bb987390be91fb6364. Latest consensus state: current_term: 1 leader_uuid: "5379a13e500442bb987390be91fb6364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5379a13e500442bb987390be91fb6364" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 36161 } } }
I20260128 20:51:58.983222 14827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:58.983651 14834 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:58.987356 14834 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:58.993894 14834 catalog_manager.cc:1357] Generated new cluster ID: 456e8479ad5f4fbd9395bfc714692aaa
I20260128 20:51:58.994043 14834 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:59.007128 14834 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:59.008271 14834 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:51:59.017323 14834 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5379a13e500442bb987390be91fb6364: Generated new TSK 0
I20260128 20:51:59.018332 14834 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:51:59.026348 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:36161
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260128 20:51:59.151670 14844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:59.151957 14844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:59.152010 14844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:59.155706 14844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:59.155795 14844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:59.155879 14844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:51:59.160286 14844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:36161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:59.161614 14844 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:59.162918 14844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:59.172376 14849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.172405 14852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.172366 14850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:59.172469 14844 server_base.cc:1060] running on GCE node
I20260128 20:51:59.173103 14844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:59.173883 14844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:59.175092 14844 hybrid_clock.cc:648] HybridClock initialized: now 1769633519175068 us; error 54 us; skew 500 ppm
I20260128 20:51:59.177536 14844 webserver.cc:492] Webserver started at http://127.4.183.129:41489/ using document root <none> and password file <none>
I20260128 20:51:59.178236 14844 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:59.178304 14844 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:59.178546 14844 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:59.180461 14844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "590cd2550c634b45a8d7880fef3f1fb7"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "f3c4030090e14a27a96cb2a379393ac9"
server_key_iv: "eb201dd32b1f798861aa3e8bc926ef5d"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.181075 14844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "590cd2550c634b45a8d7880fef3f1fb7"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "f3c4030090e14a27a96cb2a379393ac9"
server_key_iv: "eb201dd32b1f798861aa3e8bc926ef5d"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.185525 14844 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:59.188802 14858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.190562 14844 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260128 20:51:59.190821 14844 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "590cd2550c634b45a8d7880fef3f1fb7"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "f3c4030090e14a27a96cb2a379393ac9"
server_key_iv: "eb201dd32b1f798861aa3e8bc926ef5d"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.190970 14844 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:59.220008 14844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:59.221036 14844 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:59.221334 14844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:59.222147 14844 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:59.223501 14844 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:59.223587 14844 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.223677 14844 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:59.223732 14844 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.234684 14844 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:40251
I20260128 20:51:59.234704 14971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:40251 every 8 connection(s)
I20260128 20:51:59.235882 14844 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:51:59.244205 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14844
I20260128 20:51:59.244371 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:51:59.244663 4830 external_mini_cluster.cc:1468] Setting key d9ee292abacb600d83469889531310e3
I20260128 20:51:59.245666 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.238176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:57183 (local address 127.4.183.190:36161)
0128 20:51:59.238386 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:59.238390 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:59.239506 (+ 1116us) server_negotiation.cc:408] Connection header received
0128 20:51:59.240446 (+ 940us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:59.240450 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:59.240517 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:59.240632 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:59.241816 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.242470 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:59.243223 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.243413 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:59.245031 (+ 1618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:59.245066 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:59.245069 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:59.245086 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:59.245108 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:59.245116 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:59.245231 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:59.245431 (+ 200us) server_negotiation.cc:300] Negotiation successful
0128 20:51:59.245520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260128 20:51:59.246555 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.238488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:36161 (local address 127.4.183.129:57183)
0128 20:51:59.239347 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:59.239386 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:59.240220 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:59.240845 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:59.240855 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:59.240945 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:59.241652 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:59.241665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.242651 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:59.242655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:59.243088 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:59.243097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.243387 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:59.244715 (+ 1328us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:59.244717 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:59.244794 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:59.244870 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:59.245249 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:59.245259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:59.245266 (+ 7us) client_negotiation.cc:769] Sending connection context
0128 20:51:59.245430 (+ 164us) client_negotiation.cc:240] Negotiation successful
0128 20:51:59.245724 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":128,"threads_started":1}
I20260128 20:51:59.246922 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:36161
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260128 20:51:59.247849 14972 heartbeater.cc:344] Connected to a master server at 127.4.183.190:36161
I20260128 20:51:59.248178 14972 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:59.248924 14972 heartbeater.cc:507] Master 127.4.183.190:36161 requested a full tablet report, sending...
I20260128 20:51:59.250890 14784 ts_manager.cc:194] Registered new tserver with Master: 590cd2550c634b45a8d7880fef3f1fb7 (127.4.183.129:40251)
I20260128 20:51:59.252230 14784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.4.183.129:57183
W20260128 20:51:59.366117 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:59.366495 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:59.366562 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:59.371091 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:59.371192 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:59.371318 14975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:51:59.378492 14975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:36161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:59.380272 14975 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:59.382119 14975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:59.392342 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:59.392532 14975 server_base.cc:1060] running on GCE node
W20260128 20:51:59.392426 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.392349 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:59.393479 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:59.394197 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:59.395396 14975 hybrid_clock.cc:648] HybridClock initialized: now 1769633519395371 us; error 45 us; skew 500 ppm
I20260128 20:51:59.398435 14975 webserver.cc:492] Webserver started at http://127.4.183.130:43147/ using document root <none> and password file <none>
I20260128 20:51:59.399398 14975 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:59.399505 14975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:59.399783 14975 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:59.402981 14975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "4d98044992fb497aaf20a3ebe87a4da7"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "a051a4694f1951f994565a770e1ebae5"
server_key_iv: "1251e51b3a67ea8d30051c967c337ab6"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.403680 14975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "4d98044992fb497aaf20a3ebe87a4da7"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "a051a4694f1951f994565a770e1ebae5"
server_key_iv: "1251e51b3a67ea8d30051c967c337ab6"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.410482 14975 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260128 20:51:59.414189 14989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.416180 14975 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:51:59.416430 14975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "4d98044992fb497aaf20a3ebe87a4da7"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "a051a4694f1951f994565a770e1ebae5"
server_key_iv: "1251e51b3a67ea8d30051c967c337ab6"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.416586 14975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:59.436425 14975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:59.437407 14975 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:59.437664 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:59.438591 14975 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:59.440064 14975 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:59.440155 14975 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.440239 14975 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:59.440290 14975 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.451297 14975 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:45213
I20260128 20:51:59.451391 15102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:45213 every 8 connection(s)
I20260128 20:51:59.452427 14975 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:51:59.455717 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 14975
I20260128 20:51:59.455955 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:51:59.456287 4830 external_mini_cluster.cc:1468] Setting key 8a7b8e4365337bd3be7c705d243490cf
I20260128 20:51:59.458958 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:36161
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260128 20:51:59.463065 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.454825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:36245 (local address 127.4.183.190:36161)
0128 20:51:59.455306 (+ 481us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:59.455313 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:59.456460 (+ 1147us) server_negotiation.cc:408] Connection header received
0128 20:51:59.457568 (+ 1108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:59.457572 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:59.457647 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:59.457734 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:59.458922 (+ 1188us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.459515 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:59.460240 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.460426 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:59.462402 (+ 1976us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:59.462433 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:59.462435 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:59.462452 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:59.462475 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:59.462482 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:59.462570 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:59.462810 (+ 240us) server_negotiation.cc:300] Negotiation successful
0128 20:51:59.462904 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260128 20:51:59.463969 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.455400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:36161 (local address 127.4.183.130:36245)
0128 20:51:59.456253 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:59.456303 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:59.457351 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:59.457932 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:59.457946 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:59.458070 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:59.458746 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:59.458760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.459665 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:59.459668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:59.460092 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:59.460100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.460384 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:59.462075 (+ 1691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:59.462077 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:59.462124 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:59.462204 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:59.462588 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:59.462599 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:59.462613 (+ 14us) client_negotiation.cc:769] Sending connection context
0128 20:51:59.462813 (+ 200us) client_negotiation.cc:240] Negotiation successful
0128 20:51:59.463127 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":161,"threads_started":1}
I20260128 20:51:59.465332 15103 heartbeater.cc:344] Connected to a master server at 127.4.183.190:36161
I20260128 20:51:59.465638 15103 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:59.466290 15103 heartbeater.cc:507] Master 127.4.183.190:36161 requested a full tablet report, sending...
I20260128 20:51:59.467450 14784 ts_manager.cc:194] Registered new tserver with Master: 4d98044992fb497aaf20a3ebe87a4da7 (127.4.183.130:45213)
I20260128 20:51:59.468075 14784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.4.183.130:36245
W20260128 20:51:59.583298 15106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:59.583693 15106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:59.583773 15106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:59.587544 15106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:59.587633 15106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:59.587709 15106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:51:59.591982 15106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:36161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:59.593113 15106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:59.594403 15106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:59.602090 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.602116 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.602092 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:59.602222 15106 server_base.cc:1060] running on GCE node
I20260128 20:51:59.603102 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:59.603804 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:59.605010 15106 hybrid_clock.cc:648] HybridClock initialized: now 1769633519605004 us; error 43 us; skew 500 ppm
I20260128 20:51:59.607376 15106 webserver.cc:492] Webserver started at http://127.4.183.131:46131/ using document root <none> and password file <none>
I20260128 20:51:59.608047 15106 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:59.608143 15106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:59.608388 15106 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:59.610391 15106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "a66044e1fb4147f292d3eabefb986e4a"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "c427c25c1a7a611358c1d4f08110dcc6"
server_key_iv: "18e6ead5bf2867702e18b47d283b65f3"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.610968 15106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "a66044e1fb4147f292d3eabefb986e4a"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "c427c25c1a7a611358c1d4f08110dcc6"
server_key_iv: "18e6ead5bf2867702e18b47d283b65f3"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.614950 15106 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260128 20:51:59.618005 15120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.619385 15106 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:51:59.619580 15106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "a66044e1fb4147f292d3eabefb986e4a"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "c427c25c1a7a611358c1d4f08110dcc6"
server_key_iv: "18e6ead5bf2867702e18b47d283b65f3"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.619716 15106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:59.649762 15106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:59.650919 15106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:59.651278 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:59.652016 15106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:51:59.653234 15106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:51:59.653326 15106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.653409 15106 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:51:59.653456 15106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.665840 15106 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:40513
I20260128 20:51:59.665925 15233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:40513 every 8 connection(s)
I20260128 20:51:59.666940 15106 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:51:59.667656 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15106
I20260128 20:51:59.667794 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:51:59.668100 4830 external_mini_cluster.cc:1468] Setting key ee0de87630504b3972ebfedaab3af6ec
I20260128 20:51:59.679742 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.669318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:58211 (local address 127.4.183.190:36161)
0128 20:51:59.669482 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:59.669488 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:59.670674 (+ 1186us) server_negotiation.cc:408] Connection header received
0128 20:51:59.671668 (+ 994us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:59.671672 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:59.671730 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:59.671809 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:59.672864 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.673459 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:51:59.674165 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.674357 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:59.679108 (+ 4751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:59.679136 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:59.679138 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:59.679156 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:59.679186 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:59.679194 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:59.679282 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:59.679486 (+ 204us) server_negotiation.cc:300] Negotiation successful
0128 20:51:59.679578 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260128 20:51:59.680545 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.669675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:36161 (local address 127.4.183.131:58211)
0128 20:51:59.670510 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0128 20:51:59.670561 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0128 20:51:59.671439 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:51:59.671987 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:51:59.671996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:51:59.672090 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:51:59.672707 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:59.672719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.673606 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:51:59.673609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:51:59.674034 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:51:59.674043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:51:59.674321 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:51:59.678726 (+ 4405us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:51:59.678729 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:51:59.678783 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:51:59.678869 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:51:59.679302 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:51:59.679312 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:51:59.679325 (+ 13us) client_negotiation.cc:769] Sending connection context
0128 20:51:59.679496 (+ 171us) client_negotiation.cc:240] Negotiation successful
0128 20:51:59.679782 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260128 20:51:59.681865 15234 heartbeater.cc:344] Connected to a master server at 127.4.183.190:36161
I20260128 20:51:59.682156 15234 heartbeater.cc:461] Registering TS with master...
I20260128 20:51:59.682869 15234 heartbeater.cc:507] Master 127.4.183.190:36161 requested a full tablet report, sending...
I20260128 20:51:59.683923 14784 ts_manager.cc:194] Registered new tserver with Master: a66044e1fb4147f292d3eabefb986e4a (127.4.183.131:40513)
I20260128 20:51:59.684561 14784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.4.183.131:58211
I20260128 20:51:59.695761 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 20:51:59.699191 14823 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:51:59.698114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.4.183.190:36161)
0128 20:51:59.698376 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:59.698382 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:59.698487 (+ 105us) server_negotiation.cc:408] Connection header received
0128 20:51:59.698669 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:59.698672 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:59.698720 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:59.698831 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:59.699075 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60154: BlockingRecv error: recv got EOF from 127.0.0.1:60154 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
I20260128 20:51:59.699525 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14844
I20260128 20:51:59.705935 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14975
I20260128 20:51:59.711964 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15106
I20260128 20:51:59.718181 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 14753
2026-01-28T20:51:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1019 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-01-28T20:51:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:51:59Z Disabled control of system clock
I20260128 20:51:59.751246 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:44749
--webserver_interface=127.4.183.190
--webserver_port=0
--builtin_ntp_servers=127.4.183.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260128 20:51:59.870484 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:51:59.870900 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:51:59.871176 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:51:59.875471 15245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:51:59.875577 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:51:59.875595 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:51:59.875617 15245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:51:59.875635 15245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:51:59.880314 15245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:44749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:51:59.881649 15245 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:51:59.882941 15245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:51:59.889995 15251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.889995 15253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:51:59.889995 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:51:59.890151 15245 server_base.cc:1060] running on GCE node
I20260128 20:51:59.891000 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:51:59.892102 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:51:59.893254 15245 hybrid_clock.cc:648] HybridClock initialized: now 1769633519893238 us; error 36 us; skew 500 ppm
I20260128 20:51:59.895556 15245 webserver.cc:492] Webserver started at http://127.4.183.190:44449/ using document root <none> and password file <none>
I20260128 20:51:59.896581 15245 fs_manager.cc:362] Metadata directory not provided
I20260128 20:51:59.896708 15245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:51:59.896994 15245 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:51:59.899757 15245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "e28f5fc993494911b2bd153bb691ed07"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "126d5ae0b30d0df2d83adfe4e9d2d68d"
server_key_iv: "9ca231ff995704222f5b7eb21287eb15"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.900352 15245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "e28f5fc993494911b2bd153bb691ed07"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "126d5ae0b30d0df2d83adfe4e9d2d68d"
server_key_iv: "9ca231ff995704222f5b7eb21287eb15"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.904439 15245 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260128 20:51:59.907553 15259 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:51:59.909584 15245 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260128 20:51:59.909829 15245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "e28f5fc993494911b2bd153bb691ed07"
format_stamp: "Formatted at 2026-01-28 20:51:59 on dist-test-slave-hss7"
server_key: "126d5ae0b30d0df2d83adfe4e9d2d68d"
server_key_iv: "9ca231ff995704222f5b7eb21287eb15"
server_key_version: "encryptionkey@0"
I20260128 20:51:59.909997 15245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:51:59.917325 15245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:51:59.918047 15245 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:51:59.918227 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:51:59.927225 15245 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:44749
I20260128 20:51:59.927233 15311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:44749 every 8 connection(s)
I20260128 20:51:59.929284 15245 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:51:59.932873 15312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:51:59.939383 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15245
I20260128 20:51:59.939517 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:51:59.939664 15312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07: Bootstrap starting.
I20260128 20:51:59.939859 4830 external_mini_cluster.cc:1468] Setting key 384770ca992727d8f210f5cec3f8fca7
I20260128 20:51:59.942757 15312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07: Neither blocks nor log segments found. Creating new log.
I20260128 20:51:59.943758 15312 log.cc:826] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07: Log is configured to *not* fsync() on all Append() calls
I20260128 20:51:59.946441 15312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07: No bootstrap required, opened a new log
I20260128 20:51:59.946470 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:51:59.941516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.4.183.190:44749)
0128 20:51:59.942342 (+ 826us) server_negotiation.cc:207] Beginning negotiation
0128 20:51:59.942354 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0128 20:51:59.942399 (+ 45us) server_negotiation.cc:408] Connection header received
0128 20:51:59.943214 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:51:59.943245 (+ 31us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:51:59.943699 (+ 454us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:51:59.944136 (+ 437us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:51:59.945058 (+ 922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:51:59.945065 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:51:59.945078 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:51:59.945092 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:51:59.945183 (+ 91us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:51:59.945202 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:51:59.945340 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0128 20:51:59.945436 (+ 96us) server_negotiation.cc:300] Negotiation successful
0128 20:51:59.945692 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":170,"threads_started":1}
I20260128 20:51:59.949806 15312 raft_consensus.cc:359] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28f5fc993494911b2bd153bb691ed07" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 44749 } }
I20260128 20:51:59.950069 15312 raft_consensus.cc:385] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:51:59.950119 15312 raft_consensus.cc:740] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28f5fc993494911b2bd153bb691ed07, State: Initialized, Role: FOLLOWER
I20260128 20:51:59.950603 15312 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28f5fc993494911b2bd153bb691ed07" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 44749 } }
I20260128 20:51:59.950738 15312 raft_consensus.cc:399] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:51:59.950794 15312 raft_consensus.cc:493] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:51:59.950874 15312 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:51:59.951850 15312 raft_consensus.cc:515] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28f5fc993494911b2bd153bb691ed07" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 44749 } }
I20260128 20:51:59.952229 15312 leader_election.cc:304] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28f5fc993494911b2bd153bb691ed07; no voters:
I20260128 20:51:59.952590 15312 leader_election.cc:290] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:51:59.952690 15317 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:51:59.952869 15317 raft_consensus.cc:697] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [term 1 LEADER]: Becoming Leader. State: Replica: e28f5fc993494911b2bd153bb691ed07, State: Running, Role: LEADER
I20260128 20:51:59.953306 15317 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28f5fc993494911b2bd153bb691ed07" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 44749 } }
I20260128 20:51:59.954015 15312 sys_catalog.cc:565] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:51:59.955235 15318 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e28f5fc993494911b2bd153bb691ed07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28f5fc993494911b2bd153bb691ed07" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 44749 } } }
I20260128 20:51:59.955384 15318 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:59.955602 15319 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28f5fc993494911b2bd153bb691ed07. Latest consensus state: current_term: 1 leader_uuid: "e28f5fc993494911b2bd153bb691ed07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28f5fc993494911b2bd153bb691ed07" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 44749 } } }
I20260128 20:51:59.955708 15319 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07 [sys.catalog]: This master's current role is: LEADER
I20260128 20:51:59.956074 15326 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:51:59.959568 15326 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:51:59.966578 15326 catalog_manager.cc:1357] Generated new cluster ID: 13b17e7b0cd644609413bd890d37228d
I20260128 20:51:59.966792 15326 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:51:59.992563 15326 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:51:59.993006 15326 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:52:00.006896 15326 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e28f5fc993494911b2bd153bb691ed07: Generated new TSK 0
I20260128 20:52:00.007869 15326 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 20:52:00.020028 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:44749
--builtin_ntp_servers=127.4.183.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260128 20:52:00.155344 15336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:00.155643 15336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:00.155745 15336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:00.159590 15336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:00.159715 15336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:00.159839 15336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:52:00.164184 15336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:00.165510 15336 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:00.166786 15336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:00.174054 15344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:00.174078 15341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:00.174080 15342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:00.174508 15336 server_base.cc:1060] running on GCE node
I20260128 20:52:00.174935 15336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:00.175577 15336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:00.176820 15336 hybrid_clock.cc:648] HybridClock initialized: now 1769633520176792 us; error 62 us; skew 500 ppm
I20260128 20:52:00.179263 15336 webserver.cc:492] Webserver started at http://127.4.183.129:43351/ using document root <none> and password file <none>
I20260128 20:52:00.179962 15336 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:00.180065 15336 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:00.180312 15336 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:00.182324 15336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "bcf02195a612439a9d4791530055c195"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "bd28cf510094e49cb8902e7d5c862559"
server_key_iv: "a51731234c4bc7244c95136497a4e434"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.182893 15336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "bcf02195a612439a9d4791530055c195"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "bd28cf510094e49cb8902e7d5c862559"
server_key_iv: "a51731234c4bc7244c95136497a4e434"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.186870 15336 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:52:00.189936 15350 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.191282 15336 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:52:00.191473 15336 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "bcf02195a612439a9d4791530055c195"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "bd28cf510094e49cb8902e7d5c862559"
server_key_iv: "a51731234c4bc7244c95136497a4e434"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.191610 15336 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:00.197562 15336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:00.198369 15336 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:00.198591 15336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:00.199218 15336 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:00.200376 15336 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:00.200456 15336 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.200553 15336 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:00.200593 15336 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.212420 15336 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:37287
I20260128 20:52:00.212440 15463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:37287 every 8 connection(s)
I20260128 20:52:00.213591 15336 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:52:00.218282 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15336
I20260128 20:52:00.218462 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:52:00.218921 4830 external_mini_cluster.cc:1468] Setting key 9702e57b2abeceb692ba045776ac0f73
I20260128 20:52:00.220105 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:00.215639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:58913 (local address 127.4.183.190:44749)
0128 20:52:00.215802 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:00.215805 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:00.217022 (+ 1217us) server_negotiation.cc:408] Connection header received
0128 20:52:00.218037 (+ 1015us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:00.218041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:00.218095 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:00.218174 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:00.219507 (+ 1333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:00.219514 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:00.219518 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:52:00.219536 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:00.219570 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:52:00.219579 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:00.219664 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:00.219920 (+ 256us) server_negotiation.cc:300] Negotiation successful
0128 20:52:00.219966 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260128 20:52:00.221002 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:00.215986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:44749 (local address 127.4.183.129:58913)
0128 20:52:00.216842 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:00.216884 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:00.217791 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:00.218340 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:00.218349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:00.218471 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:00.219224 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:52:00.219226 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:00.219266 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:52:00.219329 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:00.219686 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:00.219695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:00.219700 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 20:52:00.219968 (+ 268us) client_negotiation.cc:240] Negotiation successful
0128 20:52:00.220222 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":121,"threads_started":1}
I20260128 20:52:00.221274 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:44749
--builtin_ntp_servers=127.4.183.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260128 20:52:00.222805 15464 heartbeater.cc:344] Connected to a master server at 127.4.183.190:44749
I20260128 20:52:00.223311 15464 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:00.224042 15464 heartbeater.cc:507] Master 127.4.183.190:44749 requested a full tablet report, sending...
I20260128 20:52:00.226080 15276 ts_manager.cc:194] Registered new tserver with Master: bcf02195a612439a9d4791530055c195 (127.4.183.129:37287)
W20260128 20:52:00.348380 15467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:00.348711 15467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:00.348819 15467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:00.352535 15467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:00.352674 15467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:00.352803 15467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:52:00.357332 15467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:00.358677 15467 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:00.359907 15467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260128 20:52:00.367373 15467 server_base.cc:1060] running on GCE node
W20260128 20:52:00.367318 15475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:00.367318 15473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:00.367579 15472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:00.368002 15467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:00.368651 15467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:00.369915 15467 hybrid_clock.cc:648] HybridClock initialized: now 1769633520369867 us; error 67 us; skew 500 ppm
I20260128 20:52:00.372254 15467 webserver.cc:492] Webserver started at http://127.4.183.130:34741/ using document root <none> and password file <none>
I20260128 20:52:00.372958 15467 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:00.373062 15467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:00.373328 15467 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:00.376544 15467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "255e32e5eb374f439a3ba5704b77f684"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "cc141ae54e0661a7299112792f103385"
server_key_iv: "3ec5acb40a96ff3f36c14ec01ea63120"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.377358 15467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "255e32e5eb374f439a3ba5704b77f684"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "cc141ae54e0661a7299112792f103385"
server_key_iv: "3ec5acb40a96ff3f36c14ec01ea63120"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.382870 15467 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260128 20:52:00.385936 15481 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.387231 15467 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:52:00.387411 15467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "255e32e5eb374f439a3ba5704b77f684"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "cc141ae54e0661a7299112792f103385"
server_key_iv: "3ec5acb40a96ff3f36c14ec01ea63120"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.387547 15467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:00.393561 15467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:00.394620 15467 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:00.394907 15467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:00.395547 15467 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:00.396670 15467 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:00.396749 15467 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.396937 15467 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:00.396998 15467 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.408140 15467 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36157
I20260128 20:52:00.408228 15594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36157 every 8 connection(s)
I20260128 20:52:00.409485 15467 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:52:00.409740 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15467
I20260128 20:52:00.409869 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:52:00.410223 4830 external_mini_cluster.cc:1468] Setting key e63e30cf642c4b8d03bb3853053a19af
I20260128 20:52:00.413440 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:44749
--builtin_ntp_servers=127.4.183.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260128 20:52:00.419062 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:00.413179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:46073 (local address 127.4.183.190:44749)
0128 20:52:00.413334 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:00.413338 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:00.414838 (+ 1500us) server_negotiation.cc:408] Connection header received
0128 20:52:00.416221 (+ 1383us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:00.416226 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:00.416296 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:00.416397 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:00.418108 (+ 1711us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:00.418112 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:00.418114 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:52:00.418131 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:00.418158 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:52:00.418167 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:00.418270 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:00.418866 (+ 596us) server_negotiation.cc:300] Negotiation successful
0128 20:52:00.418922 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 20:52:00.420811 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:00.413643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:44749 (local address 127.4.183.130:46073)
0128 20:52:00.414582 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:00.414628 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:00.415882 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:00.416660 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:00.416679 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:00.416823 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:00.417660 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:52:00.417664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:00.417729 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:52:00.417842 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:00.418282 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:00.418295 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:00.418304 (+ 9us) client_negotiation.cc:769] Sending connection context
0128 20:52:00.418950 (+ 646us) client_negotiation.cc:240] Negotiation successful
0128 20:52:00.419364 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":173,"threads_started":1}
I20260128 20:52:00.423158 15595 heartbeater.cc:344] Connected to a master server at 127.4.183.190:44749
I20260128 20:52:00.423611 15595 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:00.424466 15595 heartbeater.cc:507] Master 127.4.183.190:44749 requested a full tablet report, sending...
I20260128 20:52:00.426070 15276 ts_manager.cc:194] Registered new tserver with Master: 255e32e5eb374f439a3ba5704b77f684 (127.4.183.130:36157)
W20260128 20:52:00.546022 15598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:00.546370 15598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:00.546465 15598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:00.550482 15598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:00.550638 15598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:00.550813 15598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:52:00.556648 15598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:00.558809 15598 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:00.561350 15598 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:00.569237 15606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:00.569219 15603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:00.569293 15598 server_base.cc:1060] running on GCE node
W20260128 20:52:00.569227 15604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:00.569883 15598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:00.570515 15598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:00.571704 15598 hybrid_clock.cc:648] HybridClock initialized: now 1769633520571665 us; error 57 us; skew 500 ppm
I20260128 20:52:00.574523 15598 webserver.cc:492] Webserver started at http://127.4.183.131:38267/ using document root <none> and password file <none>
I20260128 20:52:00.575376 15598 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:00.575448 15598 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:00.575911 15598 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:00.578181 15598 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "63a8283fd4de4d3680635546eb2fbff0"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "8280aef11b3dfe70503483cd334e6f5d"
server_key_iv: "79120c17d599b3d8b92d1e7c4b7716bf"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.578778 15598 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "63a8283fd4de4d3680635546eb2fbff0"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "8280aef11b3dfe70503483cd334e6f5d"
server_key_iv: "79120c17d599b3d8b92d1e7c4b7716bf"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.583599 15598 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:52:00.586681 15612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.588699 15598 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:52:00.589057 15598 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "63a8283fd4de4d3680635546eb2fbff0"
format_stamp: "Formatted at 2026-01-28 20:52:00 on dist-test-slave-hss7"
server_key: "8280aef11b3dfe70503483cd334e6f5d"
server_key_iv: "79120c17d599b3d8b92d1e7c4b7716bf"
server_key_version: "encryptionkey@0"
I20260128 20:52:00.589435 15598 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:00.595198 15598 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:00.595918 15598 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:00.596132 15598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:00.596792 15598 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:00.598136 15598 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:00.598223 15598 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.598325 15598 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:00.598367 15598 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:00.609587 15598 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:38897
I20260128 20:52:00.609607 15727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:38897 every 8 connection(s)
I20260128 20:52:00.610729 15598 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:52:00.612138 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15598
I20260128 20:52:00.612257 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:52:00.612571 4830 external_mini_cluster.cc:1468] Setting key a8aa84db3117d45a7a1ea9e719644577
I20260128 20:52:00.617498 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:00.613217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:57337 (local address 127.4.183.190:44749)
0128 20:52:00.613493 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:00.613498 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:00.614593 (+ 1095us) server_negotiation.cc:408] Connection header received
0128 20:52:00.615572 (+ 979us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:00.615575 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:00.615621 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:00.615685 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:00.616935 (+ 1250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:00.616939 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:00.616942 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 20:52:00.616965 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:00.616987 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 20:52:00.617003 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:00.617118 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:00.617305 (+ 187us) server_negotiation.cc:300] Negotiation successful
0128 20:52:00.617354 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260128 20:52:00.618289 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:00.613491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:44749 (local address 127.4.183.131:57337)
0128 20:52:00.614428 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:00.614481 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:00.615340 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:00.615858 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:00.615867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:00.615960 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:00.616656 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 20:52:00.616658 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:00.616695 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 20:52:00.616754 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:00.617100 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:00.617108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:00.617114 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 20:52:00.617378 (+ 264us) client_negotiation.cc:240] Negotiation successful
0128 20:52:00.617624 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":162,"threads_started":1}
I20260128 20:52:00.619414 15728 heartbeater.cc:344] Connected to a master server at 127.4.183.190:44749
I20260128 20:52:00.619704 15728 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:00.620193 15728 heartbeater.cc:507] Master 127.4.183.190:44749 requested a full tablet report, sending...
I20260128 20:52:00.621434 15276 ts_manager.cc:194] Registered new tserver with Master: 63a8283fd4de4d3680635546eb2fbff0 (127.4.183.131:38897)
I20260128 20:52:00.627733 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 20:52:00.630810 15315 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 20:52:00.629944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.4.183.190:44749)
0128 20:52:00.630137 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:00.630141 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:00.630156 (+ 15us) server_negotiation.cc:408] Connection header received
0128 20:52:00.630281 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:00.630284 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:00.630323 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:00.630393 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:00.630637 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56204: BlockingRecv error: recv got EOF from 127.0.0.1:56204 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260128 20:52:00.630921 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15336
I20260128 20:52:00.637432 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15467
I20260128 20:52:00.644107 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15598
I20260128 20:52:00.650226 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15245
2026-01-28T20:52:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (932 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:52:00 dist-test-slave-hss7 krb5kdc[15738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:52:00 dist-test-slave-hss7 krb5kdc[15738](info): set up 2 sockets
Jan 28 20:52:00 dist-test-slave-hss7 krb5kdc[15738](info): commencing operation
krb5kdc: starting...
W20260128 20:52:02.699440 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:52:02 dist-test-slave-hss7 krb5kdc[15738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:52:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:02.856494 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:40583
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:52:02.980250 15754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:02.980597 15754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:02.980695 15754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:02.984500 15754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:52:02.984645 15754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:02.984694 15754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:02.984738 15754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:52:02.984789 15754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:52:02.989923 15754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:02.991219 15754 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:02.992231 15754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:02.998735 15760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:02.998991 15754 server_base.cc:1060] running on GCE node
W20260128 20:52:02.998735 15759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:02.998735 15762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:02.999678 15754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:03.000846 15754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:03.002070 15754 hybrid_clock.cc:648] HybridClock initialized: now 1769633523002044 us; error 53 us; skew 500 ppm
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:03.005199 15754 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:52:03.006524 15754 webserver.cc:492] Webserver started at http://127.4.183.190:45567/ using document root <none> and password file <none>
I20260128 20:52:03.007186 15754 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:03.007268 15754 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:03.007509 15754 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:03.009506 15754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "8a8321075373473ca5840880946154ef"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "f596a4391c518f517bae181b75cfbda2"
server_key_iv: "4ef976466c8eff4a5820c36b249127cc"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.010079 15754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "8a8321075373473ca5840880946154ef"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "f596a4391c518f517bae181b75cfbda2"
server_key_iv: "4ef976466c8eff4a5820c36b249127cc"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.014106 15754 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:52:03.016840 15769 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.018828 15754 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260128 20:52:03.019127 15754 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "8a8321075373473ca5840880946154ef"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "f596a4391c518f517bae181b75cfbda2"
server_key_iv: "4ef976466c8eff4a5820c36b249127cc"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.019301 15754 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:03.039294 15754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:03.042793 15754 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:03.043035 15754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:03.052430 15754 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:40583
I20260128 20:52:03.052430 15821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:40583 every 8 connection(s)
I20260128 20:52:03.053686 15754 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:52:03.054479 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15754
I20260128 20:52:03.054649 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:52:03.055075 4830 external_mini_cluster.cc:1468] Setting key dfbc8e13367ba57b518432315fe59788
I20260128 20:52:03.057441 15822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:52:03.064477 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef: Bootstrap starting.
I20260128 20:52:03.067770 15822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef: Neither blocks nor log segments found. Creating new log.
I20260128 20:52:03.068833 15822 log.cc:826] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef: Log is configured to *not* fsync() on all Append() calls
I20260128 20:52:03.070786 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.056388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.4.183.190:40583)
0128 20:52:03.056850 (+ 462us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.056864 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.057073 (+ 209us) server_negotiation.cc:408] Connection header received
0128 20:52:03.058414 (+ 1341us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.058439 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.058773 (+ 334us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.059187 (+ 414us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:03.060183 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.060974 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.061750 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.062079 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.064815 (+ 2736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:03.064876 (+ 61us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:03.064892 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:03.064954 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:03.067309 (+ 2355us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.068025 (+ 716us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.068033 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.068040 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.068173 (+ 133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.068442 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.068445 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.068447 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.068988 (+ 541us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:03.069264 (+ 276us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.069640 (+ 376us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.069989 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
I20260128 20:52:03.071350 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef: No bootstrap required, opened a new log
I20260128 20:52:03.074903 15822 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8321075373473ca5840880946154ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40583 } }
I20260128 20:52:03.075156 15822 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:52:03.075196 15822 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8321075373473ca5840880946154ef, State: Initialized, Role: FOLLOWER
I20260128 20:52:03.075690 15822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8321075373473ca5840880946154ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40583 } }
I20260128 20:52:03.075839 15822 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:52:03.075954 15822 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:52:03.076112 15822 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:52:03.077270 15822 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8321075373473ca5840880946154ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40583 } }
I20260128 20:52:03.077704 15822 leader_election.cc:304] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8321075373473ca5840880946154ef; no voters:
I20260128 20:52:03.078052 15822 leader_election.cc:290] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:52:03.078208 15827 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:52:03.078444 15827 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [term 1 LEADER]: Becoming Leader. State: Replica: 8a8321075373473ca5840880946154ef, State: Running, Role: LEADER
I20260128 20:52:03.078879 15827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8321075373473ca5840880946154ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40583 } }
I20260128 20:52:03.079352 15822 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:52:03.080776 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8321075373473ca5840880946154ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8321075373473ca5840880946154ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40583 } } }
I20260128 20:52:03.080946 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:52:03.080792 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8321075373473ca5840880946154ef. Latest consensus state: current_term: 1 leader_uuid: "8a8321075373473ca5840880946154ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8321075373473ca5840880946154ef" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40583 } } }
I20260128 20:52:03.081053 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef [sys.catalog]: This master's current role is: LEADER
I20260128 20:52:03.081411 15836 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:52:03.085299 15836 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:52:03.092285 15836 catalog_manager.cc:1357] Generated new cluster ID: b8498cf481d84ba18ea89a98ba38bf41
I20260128 20:52:03.092417 15836 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:52:03.107966 15836 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:52:03.109200 15836 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:52:03.120488 15836 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef: Generated new TSK 0
I20260128 20:52:03.121443 15836 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:03.190498 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:40583
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:52:03.323247 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:03.323535 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:03.323601 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:03.327939 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:03.328029 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:03.328119 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:52:03.333082 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:03.334460 15850 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:03.335407 15850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:03.343283 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:03.343241 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:03.343365 15850 server_base.cc:1060] running on GCE node
W20260128 20:52:03.343278 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:03.344231 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:03.344934 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:03.346168 15850 hybrid_clock.cc:648] HybridClock initialized: now 1769633523346141 us; error 48 us; skew 500 ppm
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:03.349488 15850 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:52:03.351058 15850 webserver.cc:492] Webserver started at http://127.4.183.129:40895/ using document root <none> and password file <none>
I20260128 20:52:03.351686 15850 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:03.351750 15850 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:03.351938 15850 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:03.354156 15850 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "26278b5babee485b9ea9918e8c7466cb"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "5a42371e74e4b6a56d68f5cb8b6a7584"
server_key_iv: "f3c0a05d153e931d5e0f2825142ada97"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.354951 15850 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "26278b5babee485b9ea9918e8c7466cb"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "5a42371e74e4b6a56d68f5cb8b6a7584"
server_key_iv: "f3c0a05d153e931d5e0f2825142ada97"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.359467 15850 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260128 20:52:03.362798 15865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.364216 15850 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:52:03.364447 15850 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "26278b5babee485b9ea9918e8c7466cb"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "5a42371e74e4b6a56d68f5cb8b6a7584"
server_key_iv: "f3c0a05d153e931d5e0f2825142ada97"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.364593 15850 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:03.391009 15850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:03.394361 15850 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:03.394594 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:03.395272 15850 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:03.396446 15850 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:03.396507 15850 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.396552 15850 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:03.396631 15850 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.407370 15850 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:45413
I20260128 20:52:03.407469 15979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:45413 every 8 connection(s)
I20260128 20:52:03.408545 15850 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:52:03.409467 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15850
I20260128 20:52:03.409583 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:52:03.409946 4830 external_mini_cluster.cc:1468] Setting key 70681d345ece9c8f4742dfe1a1405fae
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
I20260128 20:52:03.426064 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.410841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:38541 (local address 127.4.183.190:40583)
0128 20:52:03.410978 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.410983 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.411872 (+ 889us) server_negotiation.cc:408] Connection header received
0128 20:52:03.412806 (+ 934us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.412814 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.412889 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.412993 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:03.415614 (+ 2621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.416521 (+ 907us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.417700 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.418067 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.420927 (+ 2860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:03.420967 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:03.420973 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:03.421027 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:03.422839 (+ 1812us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.423588 (+ 749us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.423594 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.423597 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.423713 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.424503 (+ 790us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.424511 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.424516 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.424764 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:03.424927 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.425674 (+ 747us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.425860 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260128 20:52:03.426991 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.411185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40583 (local address 127.4.183.129:38541)
0128 20:52:03.411696 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:03.411747 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:03.412577 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:03.413324 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:03.413342 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:03.414078 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:03.415285 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.415334 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.416887 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.416891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:03.417467 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.417480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.417840 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.418545 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:52:03.418579 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:03.420653 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:03.423034 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:03.423041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:03.423054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:03.423381 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:03.423960 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:03.423968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:03.423972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:03.424190 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:03.424946 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:03.424957 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:03.425356 (+ 399us) client_negotiation.cc:769] Sending connection context
0128 20:52:03.425613 (+ 257us) client_negotiation.cc:240] Negotiation successful
0128 20:52:03.425957 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":157,"threads_started":1}
I20260128 20:52:03.428784 15980 heartbeater.cc:344] Connected to a master server at 127.4.183.190:40583
I20260128 20:52:03.429193 15980 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:03.429893 15980 heartbeater.cc:507] Master 127.4.183.190:40583 requested a full tablet report, sending...
I20260128 20:52:03.431708 15786 ts_manager.cc:194] Registered new tserver with Master: 26278b5babee485b9ea9918e8c7466cb (127.4.183.129:45413)
I20260128 20:52:03.433072 15786 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:38541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:03.473481 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:40583
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:52:03.596020 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:03.596328 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:03.596392 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:03.600325 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:03.600414 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:03.600502 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:52:03.605635 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:03.606853 15987 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:03.607748 15987 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:03.615343 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:03.615329 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:03.615329 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:03.615549 15987 server_base.cc:1060] running on GCE node
I20260128 20:52:03.616075 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:03.616731 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:03.617967 15987 hybrid_clock.cc:648] HybridClock initialized: now 1769633523617926 us; error 67 us; skew 500 ppm
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:03.621528 15987 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:52:03.622854 15987 webserver.cc:492] Webserver started at http://127.4.183.130:35299/ using document root <none> and password file <none>
I20260128 20:52:03.623502 15987 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:03.623592 15987 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:03.623827 15987 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:03.625885 15987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "6f01a430f38d4b71b81d49148a7572d3"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "36ea107354fddf1caf72a819607c2804"
server_key_iv: "0509173a651cab032b60e2ed820ba45d"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.626469 15987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "6f01a430f38d4b71b81d49148a7572d3"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "36ea107354fddf1caf72a819607c2804"
server_key_iv: "0509173a651cab032b60e2ed820ba45d"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.630388 15987 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:52:03.633229 16002 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.634663 15987 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 20:52:03.634852 15987 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "6f01a430f38d4b71b81d49148a7572d3"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "36ea107354fddf1caf72a819607c2804"
server_key_iv: "0509173a651cab032b60e2ed820ba45d"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.635002 15987 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:03.646612 15987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:03.650303 15987 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:03.650560 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:03.651258 15987 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:03.652318 15987 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:03.652400 15987 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.652473 15987 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:03.652515 15987 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.663750 15987 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:34149
I20260128 20:52:03.663769 16115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:34149 every 8 connection(s)
I20260128 20:52:03.664999 15987 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:52:03.671187 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 15987
I20260128 20:52:03.671299 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:52:03.671568 4830 external_mini_cluster.cc:1468] Setting key 1cc03a597ed7f536855882334a56022e
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:52:03.679792 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.667159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:59081 (local address 127.4.183.190:40583)
0128 20:52:03.667340 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.667344 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.668088 (+ 744us) server_negotiation.cc:408] Connection header received
0128 20:52:03.669006 (+ 918us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.669010 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.669072 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.669176 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:03.671191 (+ 2015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.671820 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.672565 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.672715 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.675363 (+ 2648us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:03.675393 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:03.675399 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:03.675448 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:03.677062 (+ 1614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.677939 (+ 877us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.677942 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.677944 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.677996 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.678441 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.678445 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.678446 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.678647 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:03.678742 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.679510 (+ 768us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.679647 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260128 20:52:03.680984 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.667411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40583 (local address 127.4.183.130:59081)
0128 20:52:03.667915 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:03.667955 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:03.668789 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:03.669356 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:03.669365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:03.669913 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:03.670985 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.671003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.671969 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.671979 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:03.672447 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.672457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.672713 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.673458 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:52:03.673489 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:03.675152 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:03.677322 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:03.677330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:03.677346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:03.677790 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:03.678117 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:03.678124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:03.678128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:03.678303 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:03.678777 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:03.678787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:03.679194 (+ 407us) client_negotiation.cc:769] Sending connection context
0128 20:52:03.679473 (+ 279us) client_negotiation.cc:240] Negotiation successful
0128 20:52:03.679891 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.4.183.131@KRBTEST.COM; defaulting to no policy
I20260128 20:52:03.682430 16116 heartbeater.cc:344] Connected to a master server at 127.4.183.190:40583
I20260128 20:52:03.682669 16116 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:03.683305 16116 heartbeater.cc:507] Master 127.4.183.190:40583 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.131@KRBTEST.COM" created.
I20260128 20:52:03.684506 15786 ts_manager.cc:194] Registered new tserver with Master: 6f01a430f38d4b71b81d49148a7572d3 (127.4.183.130:34149)
I20260128 20:52:03.685395 15786 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.130@KRBTEST.COM'} at 127.4.183.130:59081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:03.726420 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.131:0
--local_ip_for_outbound_sockets=127.4.183.131
--webserver_interface=127.4.183.131
--webserver_port=0
--tserver_master_addrs=127.4.183.190:40583
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:52:03.842206 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:03.842495 16123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:03.842590 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:03.846216 16123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:03.846356 16123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:03.846485 16123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.131
I20260128 20:52:03.851178 16123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.183.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.131
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:03.852521 16123 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:03.853662 16123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:03.861095 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:03.861095 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:03.861572 16123 server_base.cc:1060] running on GCE node
W20260128 20:52:03.861701 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:03.862530 16123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:03.863224 16123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:03.864408 16123 hybrid_clock.cc:648] HybridClock initialized: now 1769633523864394 us; error 34 us; skew 500 ppm
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:03.867506 16123 init.cc:377] Logged in from keytab as kudu/127.4.183.131@KRBTEST.COM (short username kudu)
I20260128 20:52:03.868796 16123 webserver.cc:492] Webserver started at http://127.4.183.131:46163/ using document root <none> and password file <none>
I20260128 20:52:03.869558 16123 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:03.869649 16123 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:03.869902 16123 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 20:52:03.871773 16123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/instance:
uuid: "1310e87090b04b6d994d9932491714b6"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "4e8fade5ee009116f85800ea13f4bea9"
server_key_iv: "49a77d1a24474f4fc244eaed2ffbbe69"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.872344 16123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance:
uuid: "1310e87090b04b6d994d9932491714b6"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "4e8fade5ee009116f85800ea13f4bea9"
server_key_iv: "49a77d1a24474f4fc244eaed2ffbbe69"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.876289 16123 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 20:52:03.879132 16138 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.880331 16123 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260128 20:52:03.880479 16123 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
uuid: "1310e87090b04b6d994d9932491714b6"
format_stamp: "Formatted at 2026-01-28 20:52:03 on dist-test-slave-hss7"
server_key: "4e8fade5ee009116f85800ea13f4bea9"
server_key_iv: "49a77d1a24474f4fc244eaed2ffbbe69"
server_key_version: "encryptionkey@0"
I20260128 20:52:03.880600 16123 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:03.894884 16123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:03.898115 16123 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:03.898346 16123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:03.898976 16123 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:03.899991 16123 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:03.900067 16123 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.900143 16123 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:03.900194 16123 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:03.909941 16123 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.131:37525
I20260128 20:52:03.909968 16251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.131:37525 every 8 connection(s)
I20260128 20:52:03.911057 16123 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/data/info.pb
I20260128 20:52:03.913272 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 16123
I20260128 20:52:03.913429 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/instance
I20260128 20:52:03.913757 4830 external_mini_cluster.cc:1468] Setting key 64a587cfc42abb3cd2722ac039de9483
Jan 28 20:52:03 dist-test-slave-hss7 krb5kdc[15738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633523, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.131@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:52:03.927634 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.913055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.131:35081 (local address 127.4.183.190:40583)
0128 20:52:03.913232 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.913236 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.914486 (+ 1250us) server_negotiation.cc:408] Connection header received
0128 20:52:03.915681 (+ 1195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.915685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.915770 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.915918 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:03.918517 (+ 2599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.919201 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.919994 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.920142 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.923362 (+ 3220us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:03.923391 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:03.923394 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:03.923428 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:03.925240 (+ 1812us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.925874 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.925878 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.925879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.925931 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.926308 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.926311 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.926312 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.926488 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:03.926695 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.927285 (+ 590us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.927465 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 20:52:03.928448 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.913393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40583 (local address 127.4.183.131:35081)
0128 20:52:03.914341 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:03.914394 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:03.915419 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:03.916283 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:03.916298 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:03.916958 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:03.918264 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.918283 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.919369 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.919372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:03.919823 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.919832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.920037 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.920684 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:52:03.920721 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:03.923136 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:03.925415 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:03.925422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:03.925434 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:03.925743 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:03.926020 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:03.926022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:03.926024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:03.926134 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:03.926630 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:03.926637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:03.926928 (+ 291us) client_negotiation.cc:769] Sending connection context
0128 20:52:03.927200 (+ 272us) client_negotiation.cc:240] Negotiation successful
0128 20:52:03.927576 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":148,"threads_started":1}
I20260128 20:52:03.929698 16252 heartbeater.cc:344] Connected to a master server at 127.4.183.190:40583
I20260128 20:52:03.929955 16252 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:03.930498 16252 heartbeater.cc:507] Master 127.4.183.190:40583 requested a full tablet report, sending...
I20260128 20:52:03.931533 15786 ts_manager.cc:194] Registered new tserver with Master: 1310e87090b04b6d994d9932491714b6 (127.4.183.131:37525)
I20260128 20:52:03.932111 15786 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.131@KRBTEST.COM'} at 127.4.183.131:35081
I20260128 20:52:03.940758 4830 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 20:52:03.953344 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.943706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.4.183.190:40583)
0128 20:52:03.943939 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.943944 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.943982 (+ 38us) server_negotiation.cc:408] Connection header received
0128 20:52:03.944188 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.944192 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.944260 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.944451 (+ 191us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:03.945796 (+ 1345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.946394 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.947398 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.947591 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.948786 (+ 1195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:03.948830 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:03.948832 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:03.948872 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:03.950857 (+ 1985us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.951501 (+ 644us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.951506 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.951508 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.951620 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:03.952271 (+ 651us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:03.952276 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:03.952278 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:03.952575 (+ 297us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:03.952715 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.952990 (+ 275us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.953200 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260128 20:52:03.957732 15786 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 20:52:03.960498 15786 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 20:52:03.976702 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.971872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.130:34149 (local address 127.0.0.1:51772)
0128 20:52:03.972328 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:03.972377 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:03.972553 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:03.973387 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:03.973395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:03.973427 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:52:03.973753 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.973762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.975139 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.975143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:03.976122 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.976131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.976236 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.976320 (+ 84us) client_negotiation.cc:769] Sending connection context
0128 20:52:03.976426 (+ 106us) client_negotiation.cc:240] Negotiation successful
0128 20:52:03.976537 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20260128 20:52:03.976948 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.972680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:37525 (local address 127.0.0.1:38458)
0128 20:52:03.973504 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:03.973521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:03.973612 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:03.974206 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:03.974209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:03.974228 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:52:03.974519 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.974528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.975684 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.975691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:03.976528 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.976537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.976657 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.976676 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 20:52:03.976732 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:52:03.976799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":113,"threads_started":1}
I20260128 20:52:03.977244 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.972883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:45413 (local address 127.0.0.1:38080)
0128 20:52:03.973738 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:03.973752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:03.973832 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:03.974407 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:03.974412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:03.974427 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:52:03.974644 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.974657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.975978 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.975982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:03.976824 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:03.976839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.976960 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.976976 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 20:52:03.977023 (+ 47us) client_negotiation.cc:240] Negotiation successful
0128 20:52:03.977091 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":114,"threads_started":1}
I20260128 20:52:03.978117 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.973181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.4.183.129:45413)
0128 20:52:03.973537 (+ 356us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.973544 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.973814 (+ 270us) server_negotiation.cc:408] Connection header received
0128 20:52:03.973978 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.973987 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.974156 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.974279 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:52:03.974774 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.975831 (+ 1057us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.976970 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.977616 (+ 646us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.977789 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.977878 (+ 89us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.977971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":100,"threads_started":1}
I20260128 20:52:03.978133 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.972914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.4.183.131:37525)
0128 20:52:03.973354 (+ 440us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.973360 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.973715 (+ 355us) server_negotiation.cc:408] Connection header received
0128 20:52:03.973879 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.973887 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.974061 (+ 174us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.974226 (+ 165us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:52:03.974683 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.975548 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.977073 (+ 1525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.977685 (+ 612us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.977789 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.977867 (+ 78us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.977971 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":131,"threads_started":1}
I20260128 20:52:03.979116 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:03.972120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.4.183.130:34149)
0128 20:52:03.972586 (+ 466us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:03.972596 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:03.972626 (+ 30us) server_negotiation.cc:408] Connection header received
0128 20:52:03.972726 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:03.972735 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:03.972978 (+ 243us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:03.973169 (+ 191us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:52:03.973902 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.974992 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:03.977235 (+ 2243us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:03.977881 (+ 646us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:03.978009 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:03.978091 (+ 82us) server_negotiation.cc:300] Negotiation successful
0128 20:52:03.978193 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":129,"threads_started":1}
I20260128 20:52:03.981483 15914 tablet_service.cc:1505] Processing CreateTablet for tablet 72ea3290484449d290b266686deeaabe (DEFAULT_TABLE table=test-table [id=e0cf1a22c025424fa2527d83e23ed1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:52:03.981470 16186 tablet_service.cc:1505] Processing CreateTablet for tablet 72ea3290484449d290b266686deeaabe (DEFAULT_TABLE table=test-table [id=e0cf1a22c025424fa2527d83e23ed1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:52:03.981935 16050 tablet_service.cc:1505] Processing CreateTablet for tablet 72ea3290484449d290b266686deeaabe (DEFAULT_TABLE table=test-table [id=e0cf1a22c025424fa2527d83e23ed1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 20:52:03.982616 15914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ea3290484449d290b266686deeaabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:52:03.982621 16186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ea3290484449d290b266686deeaabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:52:03.982769 16050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ea3290484449d290b266686deeaabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:52:03.990427 16271 tablet_bootstrap.cc:492] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Bootstrap starting.
I20260128 20:52:03.990432 16269 tablet_bootstrap.cc:492] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: Bootstrap starting.
I20260128 20:52:03.990754 16270 tablet_bootstrap.cc:492] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Bootstrap starting.
I20260128 20:52:03.992810 16270 tablet_bootstrap.cc:654] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Neither blocks nor log segments found. Creating new log.
I20260128 20:52:03.992820 16269 tablet_bootstrap.cc:654] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: Neither blocks nor log segments found. Creating new log.
I20260128 20:52:03.992831 16271 tablet_bootstrap.cc:654] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Neither blocks nor log segments found. Creating new log.
I20260128 20:52:03.993827 16270 log.cc:826] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Log is configured to *not* fsync() on all Append() calls
I20260128 20:52:03.994364 16271 log.cc:826] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Log is configured to *not* fsync() on all Append() calls
I20260128 20:52:03.994364 16269 log.cc:826] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: Log is configured to *not* fsync() on all Append() calls
I20260128 20:52:03.996279 16270 tablet_bootstrap.cc:492] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: No bootstrap required, opened a new log
I20260128 20:52:03.996286 16269 tablet_bootstrap.cc:492] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: No bootstrap required, opened a new log
I20260128 20:52:03.996286 16271 tablet_bootstrap.cc:492] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: No bootstrap required, opened a new log
I20260128 20:52:03.996510 16271 ts_tablet_manager.cc:1403] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260128 20:52:03.996510 16269 ts_tablet_manager.cc:1403] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 20:52:03.996515 16270 ts_tablet_manager.cc:1403] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 20:52:03.999886 16270 raft_consensus.cc:359] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:03.999924 16269 raft_consensus.cc:359] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.000151 16270 raft_consensus.cc:385] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:52:04.000161 16269 raft_consensus.cc:385] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:52:04.000217 16269 raft_consensus.cc:740] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26278b5babee485b9ea9918e8c7466cb, State: Initialized, Role: FOLLOWER
I20260128 20:52:04.000243 16270 raft_consensus.cc:740] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f01a430f38d4b71b81d49148a7572d3, State: Initialized, Role: FOLLOWER
I20260128 20:52:04.000252 16271 raft_consensus.cc:359] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.000455 16271 raft_consensus.cc:385] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:52:04.000546 16271 raft_consensus.cc:740] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1310e87090b04b6d994d9932491714b6, State: Initialized, Role: FOLLOWER
I20260128 20:52:04.000816 16270 consensus_queue.cc:260] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.000809 16269 consensus_queue.cc:260] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.000902 16271 consensus_queue.cc:260] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.001873 16269 ts_tablet_manager.cc:1434] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260128 20:52:04.001881 16271 ts_tablet_manager.cc:1434] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260128 20:52:04.001873 16270 ts_tablet_manager.cc:1434] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260128 20:52:04.002067 16252 heartbeater.cc:499] Master 127.4.183.190:40583 was elected leader, sending a full tablet report...
I20260128 20:52:04.002135 16116 heartbeater.cc:499] Master 127.4.183.190:40583 was elected leader, sending a full tablet report...
I20260128 20:52:04.003304 15980 heartbeater.cc:499] Master 127.4.183.190:40583 was elected leader, sending a full tablet report...
W20260128 20:52:04.160406 15981 tablet.cc:2378] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 20:52:04.162240 16253 tablet.cc:2378] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 20:52:04.166404 16117 tablet.cc:2378] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 20:52:04.202430 16275 raft_consensus.cc:493] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 20:52:04.202695 16275 raft_consensus.cc:515] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.203959 16275 leader_election.cc:290] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1310e87090b04b6d994d9932491714b6 (127.4.183.131:37525), 26278b5babee485b9ea9918e8c7466cb (127.4.183.129:45413)
I20260128 20:52:04.208814 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:04.204477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.131:37525 (local address 127.4.183.130:32831)
0128 20:52:04.205037 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:04.205057 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:04.205263 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:04.205629 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:04.205633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:04.205683 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:52:04.206018 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:04.206023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.207220 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:04.207225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:04.208236 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:04.208248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.208403 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:04.208429 (+ 26us) client_negotiation.cc:769] Sending connection context
0128 20:52:04.208485 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:52:04.208561 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"spinlock_wait_cycles":12160,"thread_start_us":239,"threads_started":1}
I20260128 20:52:04.208814 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:04.204477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:45413 (local address 127.4.183.130:56331)
0128 20:52:04.205102 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:04.205117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:04.205272 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:04.205629 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:04.205633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:04.205681 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 20:52:04.206013 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:04.206022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.207220 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:04.207225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:04.208236 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:04.208248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.208405 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:04.208429 (+ 24us) client_negotiation.cc:769] Sending connection context
0128 20:52:04.208485 (+ 56us) client_negotiation.cc:240] Negotiation successful
0128 20:52:04.208561 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":83,"spinlock_wait_cycles":7296,"thread_start_us":89,"threads_started":1}
I20260128 20:52:04.209281 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:04.204628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:32831 (local address 127.4.183.131:37525)
0128 20:52:04.204834 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:04.204839 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:04.205203 (+ 364us) server_negotiation.cc:408] Connection header received
0128 20:52:04.205396 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:04.205400 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:04.205464 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:04.205579 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:52:04.206194 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.207046 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:04.208430 (+ 1384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.208954 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:04.209000 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:04.209064 (+ 64us) server_negotiation.cc:300] Negotiation successful
0128 20:52:04.209158 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260128 20:52:04.209281 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:04.204807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.130:56331 (local address 127.4.183.129:45413)
0128 20:52:04.205001 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:04.205007 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:04.205218 (+ 211us) server_negotiation.cc:408] Connection header received
0128 20:52:04.205396 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:04.205400 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:04.205464 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:04.205584 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 20:52:04.206194 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.207046 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:04.208430 (+ 1384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.208960 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:04.209000 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:04.209064 (+ 64us) server_negotiation.cc:300] Negotiation successful
0128 20:52:04.209130 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260128 20:52:04.210022 15934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72ea3290484449d290b266686deeaabe" candidate_uuid: "6f01a430f38d4b71b81d49148a7572d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26278b5babee485b9ea9918e8c7466cb" is_pre_election: true
I20260128 20:52:04.210022 16206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72ea3290484449d290b266686deeaabe" candidate_uuid: "6f01a430f38d4b71b81d49148a7572d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1310e87090b04b6d994d9932491714b6" is_pre_election: true
I20260128 20:52:04.210335 15934 raft_consensus.cc:2468] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f01a430f38d4b71b81d49148a7572d3 in term 0.
I20260128 20:52:04.210335 16206 raft_consensus.cc:2468] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f01a430f38d4b71b81d49148a7572d3 in term 0.
I20260128 20:52:04.210901 16003 leader_election.cc:304] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1310e87090b04b6d994d9932491714b6, 6f01a430f38d4b71b81d49148a7572d3; no voters:
I20260128 20:52:04.211447 16275 raft_consensus.cc:2804] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 20:52:04.211545 16275 raft_consensus.cc:493] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 20:52:04.211632 16275 raft_consensus.cc:3060] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:52:04.212728 16275 raft_consensus.cc:515] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.213279 16275 leader_election.cc:290] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [CANDIDATE]: Term 1 election: Requested vote from peers 1310e87090b04b6d994d9932491714b6 (127.4.183.131:37525), 26278b5babee485b9ea9918e8c7466cb (127.4.183.129:45413)
I20260128 20:52:04.213716 16206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72ea3290484449d290b266686deeaabe" candidate_uuid: "6f01a430f38d4b71b81d49148a7572d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1310e87090b04b6d994d9932491714b6"
I20260128 20:52:04.213762 15934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72ea3290484449d290b266686deeaabe" candidate_uuid: "6f01a430f38d4b71b81d49148a7572d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26278b5babee485b9ea9918e8c7466cb"
I20260128 20:52:04.213917 15934 raft_consensus.cc:3060] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:52:04.213917 16206 raft_consensus.cc:3060] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:52:04.215471 16206 raft_consensus.cc:2468] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f01a430f38d4b71b81d49148a7572d3 in term 1.
I20260128 20:52:04.215864 15934 raft_consensus.cc:2468] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f01a430f38d4b71b81d49148a7572d3 in term 1.
I20260128 20:52:04.215894 16003 leader_election.cc:304] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1310e87090b04b6d994d9932491714b6, 6f01a430f38d4b71b81d49148a7572d3; no voters:
I20260128 20:52:04.216215 16275 raft_consensus.cc:2804] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:52:04.216490 16275 raft_consensus.cc:697] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 1 LEADER]: Becoming Leader. State: Replica: 6f01a430f38d4b71b81d49148a7572d3, State: Running, Role: LEADER
I20260128 20:52:04.216903 16275 consensus_queue.cc:237] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } }
I20260128 20:52:04.220706 15786 catalog_manager.cc:5654] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f01a430f38d4b71b81d49148a7572d3 (127.4.183.130). New cstate: current_term: 1 leader_uuid: "6f01a430f38d4b71b81d49148a7572d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f01a430f38d4b71b81d49148a7572d3" member_type: VOTER last_known_addr { host: "127.4.183.130" port: 34149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 } health_report { overall_health: UNKNOWN } } }
I20260128 20:52:04.278386 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:04.274904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.4.183.130:34149)
0128 20:52:04.275052 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:04.275056 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:04.275072 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:52:04.275167 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:04.275170 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:04.275228 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:04.275325 (+ 97us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 20:52:04.275697 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.276263 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:04.277031 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:04.277229 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:04.277341 (+ 112us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 20:52:04.277879 (+ 538us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 20:52:04.277988 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:04.278171 (+ 183us) server_negotiation.cc:300] Negotiation successful
0128 20:52:04.278256 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 20:52:04.284725 16206 raft_consensus.cc:1275] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 1 FOLLOWER]: Refusing update from remote peer 6f01a430f38d4b71b81d49148a7572d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:52:04.284745 15934 raft_consensus.cc:1275] T 72ea3290484449d290b266686deeaabe P 26278b5babee485b9ea9918e8c7466cb [term 1 FOLLOWER]: Refusing update from remote peer 6f01a430f38d4b71b81d49148a7572d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 20:52:04.285581 16280 consensus_queue.cc:1048] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1310e87090b04b6d994d9932491714b6" member_type: VOTER last_known_addr { host: "127.4.183.131" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:52:04.285840 16275 consensus_queue.cc:1048] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26278b5babee485b9ea9918e8c7466cb" member_type: VOTER last_known_addr { host: "127.4.183.129" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 20:52:04.294814 16284 mvcc.cc:204] Tried to move back new op lower bound from 7248418915462025216 to 7248418915195904000. Current Snapshot: MvccSnapshot[applied={T|T < 7248418915462025216}]
I20260128 20:52:04.308296 15784 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:52:04.308518 15784 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 20:52:04.312161 15784 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8a8321075373473ca5840880946154ef: Sending DeleteTablet for 3 replicas of tablet 72ea3290484449d290b266686deeaabe
I20260128 20:52:04.313266 16186 tablet_service.cc:1552] Processing DeleteTablet for tablet 72ea3290484449d290b266686deeaabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:52:04 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:38458
I20260128 20:52:04.313264 16050 tablet_service.cc:1552] Processing DeleteTablet for tablet 72ea3290484449d290b266686deeaabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:52:04 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:51772
I20260128 20:52:04.313869 15914 tablet_service.cc:1552] Processing DeleteTablet for tablet 72ea3290484449d290b266686deeaabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 20:52:04 UTC) from {username='kudu', principal='kudu/127.4.183.190@KRBTEST.COM'} at 127.0.0.1:38080
I20260128 20:52:04.314136 16293 tablet_replica.cc:333] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: stopping tablet replica
I20260128 20:52:04.314157 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15850
I20260128 20:52:04.314481 16293 raft_consensus.cc:2243] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 20:52:04.314870 16293 raft_consensus.cc:2272] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:52:04.315464 16294 tablet_replica.cc:333] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: stopping tablet replica
I20260128 20:52:04.315779 16294 raft_consensus.cc:2243] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 1 LEADER]: Raft consensus shutting down.
I20260128 20:52:04.316300 16294 raft_consensus.cc:2272] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 20:52:04.316700 16293 ts_tablet_manager.cc:1916] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:52:04.318179 16294 ts_tablet_manager.cc:1916] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 20:52:04.319862 16293 ts_tablet_manager.cc:1929] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:52:04.320040 16293 log.cc:1199] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-2/wal/wals/72ea3290484449d290b266686deeaabe
I20260128 20:52:04.320453 16293 ts_tablet_manager.cc:1950] T 72ea3290484449d290b266686deeaabe P 1310e87090b04b6d994d9932491714b6: Deleting consensus metadata
I20260128 20:52:04.322216 15770 catalog_manager.cc:4985] TS 1310e87090b04b6d994d9932491714b6 (127.4.183.131:37525): tablet 72ea3290484449d290b266686deeaabe (table test-table [id=e0cf1a22c025424fa2527d83e23ed1d4]) successfully deleted
I20260128 20:52:04.322271 16294 ts_tablet_manager.cc:1929] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 20:52:04.322371 16294 log.cc:1199] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Deleting WAL directory at /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769633420334841-4830-0/minicluster-data/ts-1/wal/wals/72ea3290484449d290b266686deeaabe
I20260128 20:52:04.322687 16294 ts_tablet_manager.cc:1950] T 72ea3290484449d290b266686deeaabe P 6f01a430f38d4b71b81d49148a7572d3: Deleting consensus metadata
W20260128 20:52:04.323297 15773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.183.129:45413 (error 108)
I20260128 20:52:04.324007 15770 catalog_manager.cc:4985] TS 6f01a430f38d4b71b81d49148a7572d3 (127.4.183.130:34149): tablet 72ea3290484449d290b266686deeaabe (table test-table [id=e0cf1a22c025424fa2527d83e23ed1d4]) successfully deleted
I20260128 20:52:04.324227 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15987
I20260128 20:52:04.324424 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:04.324124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.129:45413 (local address 127.0.0.1:38082)
0128 20:52:04.324251 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:04.324323 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260128 20:52:04.325773 15773 catalog_manager.cc:4712] TS 26278b5babee485b9ea9918e8c7466cb (127.4.183.129:45413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72ea3290484449d290b266686deeaabe: Network error: Client connection negotiation failed: client connection to 127.4.183.129:45413: connect: Connection refused (error 111)
I20260128 20:52:04.330664 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 16123
I20260128 20:52:04.337926 4830 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu with pid 15754
2026-01-28T20:52:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3690 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 20:52:04 dist-test-slave-hss7 krb5kdc[16299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 20:52:04 dist-test-slave-hss7 krb5kdc[16299](info): set up 2 sockets
Jan 28 20:52:04 dist-test-slave-hss7 krb5kdc[16299](info): commencing operation
krb5kdc: starting...
W20260128 20:52:06.407280 4830 mini_kdc.cc:121] Time spent starting KDC: real 2.043s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/test-user.keytab.
Jan 28 20:52:06 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T20:52:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T20:52:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/rangeradmin_127.4.183.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/rangeradmin_127.4.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/rangerlookup_127.4.183.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/rangerlookup_127.4.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/HTTP_127.4.183.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/HTTP_127.4.183.148@KRBTEST.COM.keytab.
I20260128 20:52:06.600857 4830 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMIbJnS/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/postgres -l logfile start
2026-01-28 20:52:09.593 UTC [16330] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-01-28 20:52:09.593 UTC [16330] LOG: listening on IPv4 address "127.4.183.148", port 58163
2026-01-28 20:52:09.598 UTC [16330] LOG: listening on Unix socket "/tmp/.s.PGSQL.58163"
2026-01-28 20:52:09.605 UTC [16335] LOG: database system was shut down at 2026-01-28 20:52:07 UTC
2026-01-28 20:52:09.610 UTC [16330] LOG: database system is ready to accept connections
I20260128 20:52:11.547148 4830 mini_postgres.cc:96] Postgres bound to 58163
2026-01-28 20:52:11.555 UTC [16341] FATAL: database "slave" does not exist
127.4.183.148:58163 - accepting connections
I20260128 20:52:11.555917 4830 mini_ranger.cc:161] Starting Ranger...
I20260128 20:52:11.577912 4830 mini_ranger.cc:84] Created miniranger Postgres user
I20260128 20:52:11.654219 4830 mini_ranger.cc:87] Created ranger Postgres database
I20260128 20:52:11.654325 4830 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-admin
2026-01-28 20:52:12,012 [I] DB FLAVOR :POSTGRES
2026-01-28 20:52:12,013 [I] --------- Verifying Ranger DB connection ---------
2026-01-28 20:52:12,013 [I] Checking connection..
2026-01-28 20:52:12,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-01-28 20:52:12,388 [I] Checking connection passed.
2026-01-28 20:52:12,389 [I] --------- Verifying version history table ---------
2026-01-28 20:52:12,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-28 20:52:12,744 [I] Table x_db_version_h does not exist in database ranger
2026-01-28 20:52:12,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-28 20:52:13,119 [I] Table x_db_version_h does not exist in database ranger
2026-01-28 20:52:13,120 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-01-28 20:52:13,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-01-28 20:52:13.459 UTC [16408] WARNING: there is no transaction in progress
2026-01-28 20:52:13,466 [I] create_dbversion_catalog.sql file imported successfully
2026-01-28 20:52:13,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-28 20:52:13,844 [I] Table x_db_version_h already exists in database 'ranger'
2026-01-28 20:52:13,844 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-28 20:52:13,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-01-28 20:52:14,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-01-28 20:52:14,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hss7.c.gcp-upstream.internal','N') ;"
2026-01-28 20:52:14,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-28 20:52:15,298 [I] Table x_portal_user does not exist in database ranger
2026-01-28 20:52:15,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-28 20:52:15,664 [I] Table x_policy_ref_group does not exist in database ranger
2026-01-28 20:52:15,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-28 20:52:16,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-28 20:52:16,374 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-01-28 20:52:16,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-01-28 20:52:17.411 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.423 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.436 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.449 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.463 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.478 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.534 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.540 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.552 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.564 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.575 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.588 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.597 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.604 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.609 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:17.614 UTC [16525] WARNING: there is no transaction in progress
2026-01-28 20:52:18,189 [I] ranger_core_db_postgres.sql file imported successfully
2026-01-28 20:52:18,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-01-28 20:52:18,527 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-01-28 20:52:18,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-28 20:52:18,875 [I] Table x_portal_user already exists in database 'ranger'
2026-01-28 20:52:18,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-28 20:52:19,238 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-01-28 20:52:19,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-28 20:52:19,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-28 20:52:19,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hss7.c.gcp-upstream.internal';"
2026-01-28 20:52:20,363 [I] CORE_DB_SCHEMA import status has been updated
2026-01-28 20:52:20,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-01-28 20:52:20,750 [I] DB_PATCHES have already been applied
I20260128 20:52:20.756603 4830 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMIbJnS/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260128 20:52:20.756693 4830 mini_ranger.cc:193] Using host: 127.4.183.148
I20260128 20:52:20.761044 4830 mini_ranger.cc:227] Ranger admin URL: http://127.4.183.148:35299
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-admin/ews/webapp
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 28, 2026 8:52:21 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-admin/ews/webapp].
Jan 28, 2026 8:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Jan 28, 2026 8:52:21 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-35299"]
Jan 28, 2026 8:52:21 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 28, 2026 8:52:21 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 28, 2026 8:52:22 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jan 28, 2026 8:52:22 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Jan 28, 2026 8:52:22 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Jan 28, 2026 8:52:22 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Jan 28, 2026 8:52:22 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 28, 2026 8:52:22 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
I20260128 20:52:23.039799 4830 mini_ranger.cc:160] Time spent starting Ranger: real 11.484s user 0.000s sys 0.013s
Jan 28, 2026 8:52:23 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Jan 28, 2026 8:52:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 28, 2026 8:52:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.SecurityZoneREST
Jan 28, 2026 8:52:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 28, 2026 8:52:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 28, 2026 8:52:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Jan 28, 2026 8:52:45 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-35299"]
I20260128 20:52:46.529003 4830 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/rangerkms_127.4.183.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/rangerkms_127.4.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.183.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.183.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/HTTP_127.4.183.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.183.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/HTTP_127.4.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jan 28 20:52:46 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633566, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260128 20:52:46.613370 4830 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260128 20:52:46.623617 4830 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260128 20:52:46.735580 4830 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260128 20:52:46.735704 4830 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms
I20260128 20:52:46.735754 4830 mini_ranger_kms.cc:227] Using postgres at 127.4.183.148:58163
2026-01-28 20:52:46,928 [I] DB FLAVOR :POSTGRES
2026-01-28 20:52:46,928 [I] --------- Verifying Ranger DB connection ---------
2026-01-28 20:52:46,928 [I] Checking connection
2026-01-28 20:52:46,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-01-28 20:52:47,269 [I] connection success
2026-01-28 20:52:47,270 [I] --------- Verifying Ranger DB tables ---------
2026-01-28 20:52:47,270 [I] Verifying table ranger_masterkey in database rangerkms
2026-01-28 20:52:47,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-01-28 20:52:47,600 [I] Table ranger_masterkey does not exist in database rangerkms
2026-01-28 20:52:47,600 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-28 20:52:47,600 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-01-28 20:52:47,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.183.148:58163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-01-28 20:52:47,935 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260128 20:52:48.483841 4830 mini_ranger_kms.cc:310] Created kms service
I20260128 20:52:48.754958 4830 mini_ranger_kms.cc:326] Created kudu user
I20260128 20:52:48.806972 4830 mini_ranger_kms.cc:343] Created rangerkms user
Jan 28 20:52:48 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633566, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.183.148@KRBTEST.COM
I20260128 20:52:49.241230 4830 mini_ranger_kms.cc:384] Added ranger policy
I20260128 20:52:49.241485 4830 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMIbJnS/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMIbJnS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260128 20:52:49.241518 4830 mini_ranger_kms.cc:240] Using host: 127.4.183.148
I20260128 20:52:49.245573 4830 mini_ranger_kms.cc:276] Ranger KMS PID: 16717
I20260128 20:52:49.245685 4830 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.183.148:38901
Jan 28, 2026 8:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 28, 2026 8:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 28, 2026 8:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jan 28, 2026 8:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/ews/webapp] .....
Jan 28, 2026 8:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-kms/ews/webapp].
Jan 28, 2026 8:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jan 28, 2026 8:52:51 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-38901"]
Jan 28, 2026 8:52:51 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 28, 2026 8:52:51 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 28, 2026 8:52:51 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20260128 20:52:51.496851 4830 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.883s user 0.018s sys 0.157s
I20260128 20:52:51.497035 4830 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260128 20:52:51.497195 4830 mini_ranger_kms.cc:401] 127.4.183.148:38901/kms/v1/keys
Jan 28, 2026 8:52:51 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 28 20:52:51 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769633571, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.183.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jan 28 20:52:52 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633571, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.183.148@KRBTEST.COM for HTTP/127.4.183.148@KRBTEST.COM
Jan 28 20:52:52 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633571, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.183.148@KRBTEST.COM for HTTP/127.4.183.148@KRBTEST.COM
Jan 28 20:52:55 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:55.085922 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:40923
--webserver_interface=127.4.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.183.190:40923
--ranger_config_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.183.148:38901/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:52:55.204696 16770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:55.205025 16770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:55.205121 16770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:55.208683 16770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 20:52:55.208822 16770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:55.208871 16770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:55.208930 16770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 20:52:55.208985 16770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 20:52:55.214073 16770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.183.148:38901/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.183.190:40923
--ranger_config_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.183.190:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.183.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:55.215318 16770 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:55.216261 16770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:55.222676 16776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:55.222793 16770 server_base.cc:1060] running on GCE node
W20260128 20:52:55.222677 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:55.223021 16778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:55.223570 16770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:55.224642 16770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:55.225826 16770 hybrid_clock.cc:648] HybridClock initialized: now 1769633575225817 us; error 36 us; skew 500 ppm
Jan 28 20:52:55 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633575, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:55.228750 16770 init.cc:377] Logged in from keytab as kudu/127.4.183.190@KRBTEST.COM (short username kudu)
I20260128 20:52:55.230127 16770 webserver.cc:492] Webserver started at http://127.4.183.190:33241/ using document root <none> and password file <none>
I20260128 20:52:55.230772 16770 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:55.230855 16770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:55.231092 16770 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 28 20:52:55 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633575, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.190@KRBTEST.COM for HTTP/127.4.183.148@KRBTEST.COM
I20260128 20:52:55.297703 16770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data/instance:
uuid: "f48660e165284551be0fcf047ed99ef2"
format_stamp: "Formatted at 2026-01-28 20:52:55 on dist-test-slave-hss7"
server_key: "bf14da066e58183aac5753b0c15e3d77"
server_key_iv: "76da088917b614eef2d7b719d4016678"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:55.298329 16770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal/instance:
uuid: "f48660e165284551be0fcf047ed99ef2"
format_stamp: "Formatted at 2026-01-28 20:52:55 on dist-test-slave-hss7"
server_key: "bf14da066e58183aac5753b0c15e3d77"
server_key_iv: "76da088917b614eef2d7b719d4016678"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:55.302187 16770 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:52:55.318339 16785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:55.319806 16770 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260128 20:52:55.319941 16770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal
uuid: "f48660e165284551be0fcf047ed99ef2"
format_stamp: "Formatted at 2026-01-28 20:52:55 on dist-test-slave-hss7"
server_key: "bf14da066e58183aac5753b0c15e3d77"
server_key_iv: "76da088917b614eef2d7b719d4016678"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:55.320098 16770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:55.337011 16770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:55.340240 16770 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:55.340467 16770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:55.349514 16770 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.190:40923
I20260128 20:52:55.349550 16837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.190:40923 every 8 connection(s)
I20260128 20:52:55.350667 16770 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/data/info.pb
I20260128 20:52:55.353732 16838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 20:52:55.355257 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 16770
I20260128 20:52:55.355418 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/wal/instance
I20260128 20:52:55.355713 4830 external_mini_cluster.cc:1468] Setting key 953ef02c44723210867d799aeb74175d
I20260128 20:52:55.361076 16838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2: Bootstrap starting.
I20260128 20:52:55.364274 16838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2: Neither blocks nor log segments found. Creating new log.
I20260128 20:52:55.365461 16838 log.cc:826] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2: Log is configured to *not* fsync() on all Append() calls
Jan 28 20:52:55 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:52:55.368088 16838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2: No bootstrap required, opened a new log
I20260128 20:52:55.371255 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:55.357239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.4.183.190:40923)
0128 20:52:55.357770 (+ 531us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:55.357784 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:55.357829 (+ 45us) server_negotiation.cc:408] Connection header received
0128 20:52:55.358612 (+ 783us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:55.358640 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:55.359052 (+ 412us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:55.359451 (+ 399us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:55.360584 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:55.361924 (+ 1340us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:55.362957 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:55.363292 (+ 335us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:55.366367 (+ 3075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:55.366398 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:55.366427 (+ 29us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:55.366492 (+ 65us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:55.368425 (+ 1933us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:55.368947 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:55.368953 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:55.368958 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:55.369036 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:55.369337 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:55.369341 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:55.369343 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:55.369761 (+ 418us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:55.370022 (+ 261us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:55.370269 (+ 247us) server_negotiation.cc:300] Negotiation successful
0128 20:52:55.370549 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":149,"threads_started":1}
I20260128 20:52:55.372150 16838 raft_consensus.cc:359] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48660e165284551be0fcf047ed99ef2" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40923 } }
I20260128 20:52:55.372454 16838 raft_consensus.cc:385] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 20:52:55.372556 16838 raft_consensus.cc:740] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48660e165284551be0fcf047ed99ef2, State: Initialized, Role: FOLLOWER
I20260128 20:52:55.373116 16838 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48660e165284551be0fcf047ed99ef2" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40923 } }
I20260128 20:52:55.373319 16838 raft_consensus.cc:399] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 20:52:55.373410 16838 raft_consensus.cc:493] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 20:52:55.373519 16838 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 20:52:55.374783 16838 raft_consensus.cc:515] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48660e165284551be0fcf047ed99ef2" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40923 } }
I20260128 20:52:55.375228 16838 leader_election.cc:304] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48660e165284551be0fcf047ed99ef2; no voters:
I20260128 20:52:55.375598 16838 leader_election.cc:290] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 20:52:55.375756 16843 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 20:52:55.375996 16843 raft_consensus.cc:697] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [term 1 LEADER]: Becoming Leader. State: Replica: f48660e165284551be0fcf047ed99ef2, State: Running, Role: LEADER
I20260128 20:52:55.376413 16843 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48660e165284551be0fcf047ed99ef2" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40923 } }
I20260128 20:52:55.376935 16838 sys_catalog.cc:565] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 20:52:55.378255 16845 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48660e165284551be0fcf047ed99ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48660e165284551be0fcf047ed99ef2" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40923 } } }
I20260128 20:52:55.378392 16845 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [sys.catalog]: This master's current role is: LEADER
I20260128 20:52:55.378736 16844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48660e165284551be0fcf047ed99ef2. Latest consensus state: current_term: 1 leader_uuid: "f48660e165284551be0fcf047ed99ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48660e165284551be0fcf047ed99ef2" member_type: VOTER last_known_addr { host: "127.4.183.190" port: 40923 } } }
I20260128 20:52:55.378816 16844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2 [sys.catalog]: This master's current role is: LEADER
I20260128 20:52:55.379633 16838 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260128 20:52:55.380241 16852 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 20:52:55.386264 16852 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 20:52:55.393821 16852 catalog_manager.cc:1357] Generated new cluster ID: e9d3f019cbdc4c208db040d5f88bf13c
I20260128 20:52:55.393914 16852 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 20:52:55.417465 16852 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 20:52:55.418550 16852 catalog_manager.cc:1514] Loading token signing keys...
I20260128 20:52:55.433331 16852 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f48660e165284551be0fcf047ed99ef2: Generated new TSK 0
I20260128 20:52:55.434293 16852 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:56.213378 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.183.148:38901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.129:0
--local_ip_for_outbound_sockets=127.4.183.129
--webserver_interface=127.4.183.129
--webserver_port=0
--tserver_master_addrs=127.4.183.190:40923
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 20:52:56.354645 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:56.355089 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:56.355253 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:56.359050 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:56.359179 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:56.359314 16884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.129
I20260128 20:52:56.364420 16884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.183.148:38901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.183.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:40923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.129
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:56.365770 16884 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:56.366698 16884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:56.373929 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:56.374272 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:56.374517 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:56.374877 16884 server_base.cc:1060] running on GCE node
I20260128 20:52:56.375481 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:56.376355 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:56.377554 16884 hybrid_clock.cc:648] HybridClock initialized: now 1769633576377541 us; error 37 us; skew 500 ppm
Jan 28 20:52:56 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:56.381506 16884 init.cc:377] Logged in from keytab as kudu/127.4.183.129@KRBTEST.COM (short username kudu)
I20260128 20:52:56.383176 16884 webserver.cc:492] Webserver started at http://127.4.183.129:45447/ using document root <none> and password file <none>
I20260128 20:52:56.383951 16884 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:56.384032 16884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:56.384287 16884 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 28 20:52:56 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for HTTP/127.4.183.148@KRBTEST.COM
I20260128 20:52:56.402179 16884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data/instance:
uuid: "07bcdce434274b1faecf5dd1ed84dfd4"
format_stamp: "Formatted at 2026-01-28 20:52:56 on dist-test-slave-hss7"
server_key: "9763212383a2a712b00173b2fd9b55bc"
server_key_iv: "218cb8001e2bcdf0fc1668dc22ef88dd"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:56.403357 16884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance:
uuid: "07bcdce434274b1faecf5dd1ed84dfd4"
format_stamp: "Formatted at 2026-01-28 20:52:56 on dist-test-slave-hss7"
server_key: "9763212383a2a712b00173b2fd9b55bc"
server_key_iv: "218cb8001e2bcdf0fc1668dc22ef88dd"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:56.422936 16884 fs_manager.cc:696] Time spent creating directory manager: real 0.019s user 0.004s sys 0.000s
I20260128 20:52:56.442776 16901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:56.444267 16884 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:52:56.444466 16884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal
uuid: "07bcdce434274b1faecf5dd1ed84dfd4"
format_stamp: "Formatted at 2026-01-28 20:52:56 on dist-test-slave-hss7"
server_key: "9763212383a2a712b00173b2fd9b55bc"
server_key_iv: "218cb8001e2bcdf0fc1668dc22ef88dd"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:56.444597 16884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:56.466185 16884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:56.470172 16884 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:56.470487 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:56.471210 16884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:56.472460 16884 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:56.472576 16884 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260128 20:52:56.472707 16884 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:56.472784 16884 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:56.487614 16884 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.129:39683
I20260128 20:52:56.489073 16884 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/data/info.pb
I20260128 20:52:56.490195 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 16884
I20260128 20:52:56.490347 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-0/wal/instance
I20260128 20:52:56.490667 4830 external_mini_cluster.cc:1468] Setting key bd490b09a9888d389a2b5998d7b17f96
I20260128 20:52:56.491564 17016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.129:39683 every 8 connection(s)
Jan 28 20:52:56 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.129@KRBTEST.COM for kudu/127.4.183.190@KRBTEST.COM
I20260128 20:52:56.513108 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:56.493725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.129:49833)
0128 20:52:56.497595 (+ 3870us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:56.497654 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0128 20:52:56.498723 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 20:52:56.499855 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 20:52:56.499869 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 20:52:56.500288 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0128 20:52:56.501181 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:56.501201 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:56.503319 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 20:52:56.503324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 20:52:56.503965 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 20:52:56.503980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 20:52:56.504266 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:56.505119 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 20:52:56.505201 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0128 20:52:56.506991 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 20:52:56.509705 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:56.509712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:56.509729 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:56.510120 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:56.510506 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 20:52:56.510509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 20:52:56.510511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 20:52:56.510675 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 20:52:56.511247 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 20:52:56.511255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 20:52:56.511629 (+ 374us) client_negotiation.cc:769] Sending connection context
0128 20:52:56.511869 (+ 240us) client_negotiation.cc:240] Negotiation successful
0128 20:52:56.512184 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3581,"thread_start_us":145,"threads_started":1}
I20260128 20:52:56.514024 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:56.494600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.183.129:49833 (local address 127.4.183.190:40923)
0128 20:52:56.499260 (+ 4660us) server_negotiation.cc:207] Beginning negotiation
0128 20:52:56.499266 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 20:52:56.499282 (+ 16us) server_negotiation.cc:408] Connection header received
0128 20:52:56.499328 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 20:52:56.499331 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 20:52:56.499392 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 20:52:56.499498 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0128 20:52:56.501403 (+ 1905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:56.502226 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 20:52:56.504685 (+ 2459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 20:52:56.504900 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 20:52:56.507240 (+ 2340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 20:52:56.507271 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 20:52:56.507275 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 20:52:56.507312 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 20:52:56.509499 (+ 2187us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:56.510271 (+ 772us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:56.510276 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:56.510278 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:56.510378 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 20:52:56.510790 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 20:52:56.510794 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 20:52:56.510796 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 20:52:56.511077 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 20:52:56.511182 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0128 20:52:56.513676 (+ 2494us) server_negotiation.cc:300] Negotiation successful
0128 20:52:56.513846 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4549,"thread_start_us":89,"threads_started":1}
I20260128 20:52:56.516047 17017 heartbeater.cc:344] Connected to a master server at 127.4.183.190:40923
I20260128 20:52:56.516371 17017 heartbeater.cc:461] Registering TS with master...
I20260128 20:52:56.517058 17017 heartbeater.cc:507] Master 127.4.183.190:40923 requested a full tablet report, sending...
I20260128 20:52:56.519016 16802 ts_manager.cc:194] Registered new tserver with Master: 07bcdce434274b1faecf5dd1ed84dfd4 (127.4.183.129:39683)
I20260128 20:52:56.520709 16802 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.183.129@KRBTEST.COM'} at 127.4.183.129:49833
WARNING: no policy specified for kudu/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.183.130@KRBTEST.COM" created.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.4.183.190 from keytab /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.4.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.183.130@KRBTEST.COM" created.
I20260128 20:52:56.569250 16876 server.cc:273] Received an EOF from the subprocess
F20260128 20:52:56.570299 16862 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f0b10a5adcd google::LogMessage::Fail() at ??:0
@ 0x7f0b10a5eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0b10a5a7cc google::LogMessage::Flush() at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.4.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab.
I20260128 20:52:56.579952 4830 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
/tmp/dist-test-taskMIbJnS/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.183.148:38901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.183.130:0
--local_ip_for_outbound_sockets=127.4.183.130
--webserver_interface=127.4.183.130
--webserver_port=0
--tserver_master_addrs=127.4.183.190:40923
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.4.183.148:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7f0b10a5bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0b127122d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f0b1270f0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f0b12713005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555595dfda0a std::function<>::operator()() at ??:0
@ 0x7f0b11eb4b04 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0b1274e6db start_thread at ??:0
@ 0x7f0b0fc1171f clone at ??:0
W20260128 20:52:56.748039 17027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 20:52:56.748420 17027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 20:52:56.748519 17027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 20:52:56.753959 17027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 20:52:56.754071 17027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 20:52:56.754204 17027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.183.130
I20260128 20:52:56.760883 17027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.183.148:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.183.148:38901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/krb5kdc/kudu.keytab
--principal=kudu/127.4.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.183.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.4.183.190:40923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.183.130
--log_dir=/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 20:43:11 UTC on 24a791456cd2
build id 10234
I20260128 20:52:56.762446 17027 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 20:52:56.763617 17027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 20:52:56.770860 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:56.770882 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 20:52:56.771070 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 20:52:56.771806 17027 server_base.cc:1060] running on GCE node
I20260128 20:52:56.772238 17027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 20:52:56.772902 17027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 20:52:56.774070 17027 hybrid_clock.cc:648] HybridClock initialized: now 1769633576774045 us; error 39 us; skew 500 ppm
Jan 28 20:52:56 dist-test-slave-hss7 krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 20:52:56.777104 17027 init.cc:377] Logged in from keytab as kudu/127.4.183.130@KRBTEST.COM (short username kudu)
I20260128 20:52:56.778529 17027 webserver.cc:492] Webserver started at http://127.4.183.130:39989/ using document root <none> and password file <none>
I20260128 20:52:56.779129 17027 fs_manager.cc:362] Metadata directory not provided
I20260128 20:52:56.779218 17027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 20:52:56.779430 17027 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 28 20:52:56 dist-test-slave-hss7 krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769633576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.183.130@KRBTEST.COM for HTTP/127.4.183.148@KRBTEST.COM
I20260128 20:52:56.794016 17027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data/instance:
uuid: "de6e5205b68543e8953428593a09f6ca"
format_stamp: "Formatted at 2026-01-28 20:52:56 on dist-test-slave-hss7"
server_key: "66a402991e167871f49f18c4f24957b2"
server_key_iv: "4104576fecb36e0c6425f10910dc15b3"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:56.794646 17027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance:
uuid: "de6e5205b68543e8953428593a09f6ca"
format_stamp: "Formatted at 2026-01-28 20:52:56 on dist-test-slave-hss7"
server_key: "66a402991e167871f49f18c4f24957b2"
server_key_iv: "4104576fecb36e0c6425f10910dc15b3"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:56.798578 17027 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260128 20:52:56.815106 17052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:56.816606 17027 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260128 20:52:56.816768 17027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal
uuid: "de6e5205b68543e8953428593a09f6ca"
format_stamp: "Formatted at 2026-01-28 20:52:56 on dist-test-slave-hss7"
server_key: "66a402991e167871f49f18c4f24957b2"
server_key_iv: "4104576fecb36e0c6425f10910dc15b3"
server_key_version: "kuduclusterkey@0"
I20260128 20:52:56.816915 17027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 20:52:56.843788 17027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 20:52:56.850739 17027 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 20:52:56.850968 17027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 20:52:56.851790 17027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 20:52:56.853329 17027 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 20:52:56.853400 17027 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:56.853472 17027 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 20:52:56.853504 17027 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 20:52:56.864900 17027 rpc_server.cc:307] RPC server started. Bound to: 127.4.183.130:36223
I20260128 20:52:56.865398 17165 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.183.130:36223 every 8 connection(s)
I20260128 20:52:56.866146 17027 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/data/info.pb
I20260128 20:52:56.868564 4830 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskMIbJnS/build/debug/bin/kudu as pid 17027
I20260128 20:52:56.868713 4830 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskMIbJnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769633420334841-4830-0/minicluster-data/ts-1/wal/instance
I20260128 20:52:56.868994 4830 external_mini_cluster.cc:1468] Setting key 4c8e28b3343c525bdeb532eed8637d98
I20260128 20:52:56.870429 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:56.868397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:36099)
0128 20:52:56.869502 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:56.869635 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":163,"threads_started":1}
W20260128 20:52:56.870893 17166 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:40923 (0 consecutive failures): Network error: Failed to ping master at 127.4.183.190:40923: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
I20260128 20:52:56.871583 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:56.871298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:41985)
0128 20:52:56.871444 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:56.871505 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260128 20:52:56.872303 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:56.872044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:51637)
0128 20:52:56.872172 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:56.872226 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260128 20:52:56.872586 17166 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260128 20:52:57.524256 17017 heartbeater.cc:499] Master 127.4.183.190:40923 was elected leader, sending a full tablet report...
I20260128 20:52:57.525529 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:57.524857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.129:49721)
0128 20:52:57.525281 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:57.525374 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
W20260128 20:52:57.525928 17017 heartbeater.cc:646] Failed to heartbeat to 127.4.183.190:40923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
I20260128 20:52:57.873973 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:57.873471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:48683)
0128 20:52:57.873788 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:57.873853 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260128 20:52:58.527110 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:58.526658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.129:57277)
0128 20:52:58.526947 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:58.527011 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260128 20:52:58.875452 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:58.874939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:35127)
0128 20:52:58.875259 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:58.875322 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
I20260128 20:52:59.528620 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:59.528072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.129:58727)
0128 20:52:59.528447 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:59.528517 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260128 20:52:59.877105 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:52:59.876605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:59237)
0128 20:52:59.876913 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0128 20:52:59.876980 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260128 20:53:00.530050 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:53:00.529605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.129:51679)
0128 20:53:00.529881 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0128 20:53:00.529944 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260128 20:53:00.878468 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:53:00.877997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:55657)
0128 20:53:00.878312 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0128 20:53:00.878368 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260128 20:53:01.531591 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:53:01.531064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.129:49401)
0128 20:53:01.531420 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0128 20:53:01.531488 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260128 20:53:01.879864 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 20:53:01.879400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.183.190:40923 (local address 127.4.183.130:34045)
0128 20:53:01.879689 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0128 20:53:01.879754 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.183.190:40923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
F20260128 20:53:01.923112 4830 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1769633581 (unix time) try "date -d @1769633581" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000012de) received by PID 4830 (TID 0x7f6063b13d80) from PID 4830; stack trace: ***
@ 0x7f606f7ba980 (unknown) at ??:0
@ 0x7f606cdf0fb7 gsignal at ??:0
@ 0x7f606cdf2921 abort at ??:0
@ 0x7f606e05edcd google::LogMessage::Fail() at ??:0
@ 0x7f606e062b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f606e05e7cc google::LogMessage::Flush() at ??:0
@ 0x7f606e05ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6070debcab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f6070de89c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x5618acb4592f kudu::SecurityITest::StartCluster() at ??:0
@ 0x5618acb2d792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f606f06f0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f606f063bdb testing::Test::Run() at ??:0
@ 0x7f606f063d9d testing::TestInfo::Run() at ??:0
@ 0x7f606f064377 testing::TestSuite::Run() at ??:0
@ 0x7f606f06477c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f606f06f60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f606f063e63 testing::UnitTest::Run() at ??:0
@ 0x5618acb7dc7e RUN_ALL_TESTS() at ??:0
@ 0x5618acb7ce6b main at ??:0
@ 0x7f606cdd3bf7 __libc_start_main at ??:0
@ 0x5618acb1506a _start at ??:0
2026-01-28 20:53:02.126 UTC [16748] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 20:53:02.126 UTC [16641] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 20:53:02.126 UTC [16644] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 20:53:02.127 UTC [16642] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 20:53:02.128 UTC [16645] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 20:53:02.128 UTC [16640] FATAL: terminating connection due to unexpected postmaster exit